Building a Self-Running 5-Source Intelligence Pipeline: What Actually Went Wrong
Moltalyzer Team · March 2026 · 12 min read
Editor’s note: the multi-source feeds described here moved to sibling products as of 2026-07-10 — Polymarket to OrcaTrace (orcatrace.dev), GitHub to gitBeacon (gitbeacon.dev), and Master Intelligence + Pulse to Signalis (signalis.dev). Moltalyzer now serves Moltbook community intelligence. The API-key and email authentication described below has also been retired: Moltalyzer is x402-only — pay per request with USDC on Base, no accounts and no keys. This post is kept for its build history.
We built a system that continuously ingests data from five independent sources—a community forum scraper, Polymarket prediction markets, on-chain DexScreener token data, GitHub’s Search API, and a cross-source narrative tracker—synthesizes them into a structured JSON digest every hour using Claude Opus, and is managed by a five-agent autonomous team that monitors health, audits quality, executes engineering tasks, scouts new sources, and distributes content across eight channels. The whole thing runs on one Linux server as systemd units, costs $3–5/day in LLM calls, and has been running continuously for several weeks.
This post is about the technical problems we ran into, not the product. Specifically: why Llama 70B failed at captcha solving 80% of the time, how we cut prompt length by 78% without losing signal, why we had to use a WASM Prisma engine instead of the native binary, and what it took to get an autonomous agent team to not crash itself every few hours.
The five sources and their scan cadences
Each source is a separate service with its own database models, deduplication logic, and failure modes. They share a PostgreSQL database but are otherwise decoupled. If one source crashes, the hourly digest degrades gracefully instead of failing entirely.
Problem 1: Captcha solving at 100% accuracy or not at all
The community forum we scrape (Moltbook) has a verification system on every write operation. Every post and comment submission returns a 201 with a challenge object instead of a success response. You have 30 seconds to solve it and POST the answer to /api/v1/verify before the write is committed.
The challenge format is an obfuscated math problem. Letters in words are mixed case with extra characters inserted, special chars, and extra spaces. Something like:
{
"verification_required": true,
"verification": {
"challenge": "R]eD] lOoObB-sTtErS^ cLaAw] fO^rCe Is Th-iRtY TwO N-eW/tOnS + BlUe] lOoBbS wEiGhT tWeNtY NeWtOnS",
"expires_at": "2026-03-12T14:23:42Z",
"verification_id": "v_abc123"
}
}
// Decoded: "red lobsters claw force is thirty two newtons + blue lobbs weight twenty newtons"
// Answer: "52.00"Always lobster-themed arithmetic. Two or more numbers, addition or subtraction of “Newtons.” Answer must be a float with 2 decimal places.
We initially used Llama 70B (our fast-tier model via OpenRouter) to solve these. 20% success rate. Llama would either fail to parse the obfuscation or produce the right number in the wrong format or confabulate entirely. After 10+ consecutive failures, the platform suspended our agent account for 24 hours.
Switching to GPT-4o gave us 100% success rate across hundreds of challenges. Total solve time is about 600ms. The difference is not intelligence—it is instruction following. GPT-4o consistently applies the format constraint (“return only the number as X.XX”). Llama keeps including explanatory text.
async function solveCaptcha(challenge: string): Promise<string> {
// MUST use GPT-4o (complex tier). Llama 70B has 20% success rate.
const response = await llm.complete({
tier: "complex", // routes to GPT-4o via OpenRouter
prompt: `Decode this obfuscated lobster math challenge and return ONLY the numeric answer as X.XX (no text, no explanation):
Challenge: ${challenge}
Rules:
- Mixed case letters with extra chars/spaces are noise — extract the words
- Numbers are spelled out in English
- Operations are + or -
- Return ONLY the decimal answer like "52.00"`,
});
// Extract float from response regardless of any stray text
const match = response.match(/\d+\.\d{2}/);
return match ? match[0] : response.trim();
}The regex extraction at the end is a belt-and-suspenders measure. Even when GPT-4o adds a stray character, we pull the float out reliably. This pattern—strong instruction + regex extraction—is now the standard for all our structured LLM outputs.
Problem 2: Prompt pre-aggregation (30K → 6.5K chars)
Early versions of the hourly digest sent raw scraped content directly into the synthesis prompt. A typical prompt was 30,000–35,000 characters. LLMs return truncated JSON at that length—not malformed JSON, just cut off mid-object. The JSON parser succeeds until it hits the truncation point, then throws.
We tried: increasing max_tokens, switching to models with 200K context windows, splitting the prompt into chunks. None of these worked well. The real problem was that most of the prompt content was redundant—the same story appearing across multiple sources in slightly different words.
The fix was a pre-aggregation step that compresses source data into topic clusters with counts before anything hits the synthesis LLM:
// BEFORE: raw items sent directly to LLM — 30K+ chars
const rawPrompt = items.map(item =>
`[${item.source}] ${item.title}: ${item.summary}`
).join("\n");
// AFTER: cluster by topic first, send counts not full text
interface TopicCluster {
topic: string;
count: number;
sources: string[]; // which sources covered this
keyPoints: string[]; // top 3 unique points extracted
sentiment: "bullish" | "bearish" | "neutral";
}
async function preAggregate(items: ContentItem[]): Promise<TopicCluster[]> {
// Step 1: Extract topics from each item (fast-tier LLM, cheap)
const tagged = await Promise.all(
items.map(item => extractTopic(item))
);
// Step 2: Group by topic with fuzzy matching
const clusters = groupByTopic(tagged);
// Step 3: For each cluster, extract key points (not full content)
return clusters.map(cluster => ({
topic: cluster.name,
count: cluster.items.length,
sources: [...new Set(cluster.items.map(i => i.source))],
keyPoints: extractKeyPoints(cluster.items, 3),
sentiment: voteSentiment(cluster.items),
}));
}
// Result: prompt goes from ~30K chars to ~6.5K chars (78% reduction)
// Quality actually improved — LLM sees structure, not noiseThe 78% reduction was not just a cost saving. Output quality improved measurably. The synthesis LLM produces more coherent cross-domain narratives when it is working with structured topic clusters instead of wading through redundant raw text. The quality auditor agent (more on that below) scored digests 20-30% higher after this change.
One unexpected benefit: the pre-aggregation step itself surfaces signal. When 12 items from different sources all cluster into the same topic, that is a stronger signal than any single item. The cluster count is now a first-class field in the digest output.
Problem 3: Polymarket confidence scoring without volume data
Polymarket’s API returns prediction market events with probability data, but raw probability is a weak signal without volume context. A market at 0.72 probability means something very different with $2M in volume versus $8K.
We implemented volume-weighted confidence scoring with configurable thresholds:
// Environment-configurable thresholds
const VOLUME_HIGH = parseInt(process.env.POLYMARKET_VOLUME_HIGH ?? "50000");
const VOLUME_MEDIUM = parseInt(process.env.POLYMARKET_VOLUME_MEDIUM ?? "10000");
function scorePolymarketSignal(market: PolymarketEvent): Signal {
const baseConfidence = market.probability;
// Volume adjusts confidence tier, not raw score
let confidenceTier: "high" | "medium" | "low";
if (market.volume24h >= VOLUME_HIGH) {
// High volume: upgrade medium→high
confidenceTier = baseConfidence >= 0.5 ? "high" : "medium";
} else if (market.volume24h >= VOLUME_MEDIUM) {
// Medium volume: use probability directly
confidenceTier = baseConfidence >= 0.7 ? "high"
: baseConfidence >= 0.5 ? "medium" : "low";
} else {
// Low volume: downgrade medium→low (can't trust the probability)
confidenceTier = baseConfidence >= 0.8 ? "medium" : "low";
}
return {
market: market.question,
probability: baseConfidence,
volume24h: market.volume24h,
confidenceTier,
// Only surface high/medium in the digest feed
include: confidenceTier !== "low",
};
}
// API supports filtering: GET /api/signals?minVolume=50000The volume thresholds are environment variables because they needed frequent adjustment. Markets that are illiquid before resolution often spike in volume in the final 12 hours, which would temporarily upgrade their confidence tier at exactly the wrong time. We added a time-to-resolution decay that softens the volume boost as markets approach their close date.
Problem 4: Multi-LLM failover and the empty array bug
We route LLM calls through OpenRouter across three tiers: fast (Llama 70B), medium (Claude Sonnet), complex (GPT-4o). Each tier has a fallback chain. When a provider returns an error or rate-limit, the chain tries the next provider with a 2-second backoff.
The backoff delay (originally instant retry) was added after we discovered that fast retries on 429s were triggering secondary rate limits. Two seconds is enough for most provider windows to reset.
The harder problem was GPT-4o returning syntactically valid but semantically empty responses. About 27% of calls to GPT-4o would return a valid JSON object that was just empty arrays:
// What we expected:
{
"signals": [
{ "topic": "...", "confidence": 0.8, "sources": ["polymarket", "github"] }
],
"narratives": [...]
}
// What GPT-4o returned ~27% of the time:
{
"signals": [],
"narratives": []
}
// Valid JSON. No error. Completely useless.We never found the root cause. Our hypothesis is that it correlates with certain prompt structures that the model treats as a “nothing to report” case. Adding explicit instructions (“you must include at least one signal if input data is present”) reduced but did not eliminate it.
The fix is pragmatic: detect empty arrays after parsing, retry up to 3 times with a 2s delay, then fall back to a different provider if all retries produce empty results. The retry budget adds ~6 seconds of latency on affected calls but the alternative is silently serving empty digests.
async function callWithRetry<T>(
prompt: string,
tier: LLMTier,
validate: (result: T) => boolean,
maxRetries = 3,
): Promise<T> {
let lastResult: T | null = null;
for (let attempt = 0; attempt < maxRetries; attempt++) {
const result = await llmChain.complete<T>({ tier, prompt });
if (validate(result)) {
return result;
}
lastResult = result;
logger.warn(`LLM returned invalid result (attempt ${attempt + 1})`, result);
if (attempt < maxRetries - 1) {
await sleep(2000); // 2s backoff before retry
}
}
// All retries failed — escalate to next tier
if (tier !== "complex") {
return callWithRetry(prompt, "complex", validate, 2);
}
throw new Error(`LLM validation failed after ${maxRetries} retries`);
}
// Usage:
const digest = await callWithRetry(
synthesisPrompt,
"medium",
(result) => result.signals?.length > 0 && result.narratives?.length > 0,
);Problem 5: SIGSEGV from the Prisma native binary on Node.js v24
This one cost us a full day. Node.js v24 was installed on our server. The Prisma query engine ships as a native Rust binary compiled against glibc. On Node.js v24 + Linux, the native engine was crashing with SIGSEGV during certain query patterns—no error thrown to application code, just a process kill.
The fix was switching to Prisma’s WASM engine, which runs inside the Node.js runtime instead of as a separate native process. No SIGSEGV, identical API surface, negligible performance difference for our query patterns:
// packages/database/src/client.ts
import { PrismaPg } from "@prisma/adapter-pg";
import { PrismaClient } from "./generated/prisma";
import pg from "pg";
export function createPrismaClient() {
const pool = new pg.Pool({
connectionString: process.env.DATABASE_URL,
});
const adapter = new PrismaPg(pool);
// WASM engine: no native binary, no SIGSEGV on Node.js v24
return new PrismaClient({ adapter });
}
// schema.prisma change required:
// generator client {
// provider = "prisma-client-js"
// previewFeatures = ["driverAdapters"]
// }We also had to use Prisma v6 specifically. The global install was v7 which broke the datasource URL configuration. The workaround is always invoking it as npx --package=prisma@6 prisma generate instead of npx prisma generate. This is documented in our CLAUDE.md so agents don’t regress it.
Token scoring: from 31% to 37.7% win rate across 2,194 backtests
The token intelligence system scans DexScreener every 4 minutes, deduplicates against previously-seen tokens, and scores each new token on a 100-point hybrid system: 70% deterministic rules, 30% LLM analysis.
interface TokenScore {
rules: number; // 0-100, deterministic
llm: number; // 0-100, LLM analysis
hybrid: number; // weighted combination
confidenceTier: "high" | "medium" | "low";
}
function computeHybridScore(
rulesScore: number,
llmScore: number,
llmConfidence: "high" | "medium" | "low",
): TokenScore {
// Dynamic weighting based on LLM confidence
const weights = {
high: { rules: 0.60, llm: 0.40 },
medium: { rules: 0.70, llm: 0.30 },
low: { rules: 0.85, llm: 0.15 },
};
const w = weights[llmConfidence];
const hybrid = rulesScore * w.rules + llmScore * w.llm;
return { rules: rulesScore, llm: llmScore, hybrid, confidenceTier: llmConfidence };
}
// Rule categories (100pts total):
// Liquidity: 18pts — strongest single predictor ($50K+ gate)
// Transactions: 17pts — buy/sell count and ratio
// Social: 15pts — Twitter activity, verified contract
// Metadata: 15pts — name, symbol, description quality
// Volume: 13pts — 24hr volume relative to liquidity
// Price action: 10pts — price stability, not pump pattern
// Age: 12pts — newer tokens score higherThe most important finding from 2,194 backtests: $50K minimum liquidity is the strongest single win-rate predictor. Tokens below $50K liquidity have a 31.2% win rate (baseline). Above $50K: 50.9% win rate. We added it as a hard gate—tokens below $50K liquidity are not scored or classified at all.
Solana was dropped entirely. After 36 days of data, Solana tokens showed a 21.7% win rate versus 57.1% for Ethereum and 51.7% for Base. The high-velocity Solana token market is dominated by coordinated pump patterns that score well on rules but rarely sustain. Including it was adding noise that degraded overall system accuracy.
Current best numbers: high-confidence tier hits 37.7% win rate. Baseline is 31.2%. That delta sounds small, but it compounds significantly over a backtest portfolio. We are not claiming this is good enough to trade on—it is not. It is useful as one input signal among many.
The agent team that runs the system
The hardest part of building this was not the data pipeline. It was building an autonomous agent team to manage the pipeline without requiring constant human intervention. Five agents run as systemd timers:
Each agent has a persistent memory file at agents/memory/{agent-name}.json. Agents load memory at the start of each run and save learnings at the end. This lets them avoid repeating failed approaches across runs without requiring a human to intervene.
There was a bug where agent subprocesses were inheriting the CLAUDECODE environment variable and behaving oddly. The fix is delete env.CLAUDECODE—setting it to an empty string does not have the same effect. This is now documented as a MANDATORY rule for all agent implementations.
Posting to X (Twitter) without their official API
We use agent-twitter-client, a scraper-based Twitter library with tweet sending support. The distribution engine does not POST directly—it sends to an internal HTTP endpoint on our sniffer service which handles the actual Twitter session. This design means only one service maintains the authenticated session, reducing cookie invalidation risk.
Cookie persistence was non-obvious. The session needs to be saved and restored across restarts. Key lesson: never call logout() on shutdown. Logging out invalidates the session and forces a fresh login on the next restart, which triggers Cloudflare captchas. We learned this the hard way.
Media uploads required bypassing Cloudflare. Standard fetch() for the chunked upload endpoint was blocked. The fix: CycleTLS (a TLS fingerprint-spoofing HTTP client) with base64-encoded media_data in the request body. This was the third approach we tried. The first two (fetch, axios) were both blocked at the Cloudflare layer before reaching Twitter’s servers.
What does not work well yet
Cross-domain narrative synthesis is about 70% accurate. The synthesis LLM sometimes connects unrelated events as the “same story” (hallucination), and sometimes misses genuine connections between sources. Without ground truth labels, we are measuring this by having the quality auditor flag obviously wrong connections. We think we can improve it with better entity extraction, but it requires building an entity resolution layer which is a significant project.
The GitHub source has a 24-hour lag.We scan yesterday’s repositories because GitHub’s star count does not stabilize until 1-2 hours after creation. If a repo explodes on HN today, we will not have it in the GitHub signal until tomorrow. The community forum scanner (which monitors HN) catches it within 5 minutes—the GitHub source is meant to provide enriched metadata about the repository itself, not break the news.
The engineer agent completes about 60% of tasks. Complex multi-file tasks often hit edge cases or build failures that the agent cannot self-correct within a single run. We handle this by having the engineer break complex tasks into subtasks and file them to the queue, but the breakdown logic is not always correct. A task that should be three subtasks sometimes gets filed as one impossible task.
Token win rates are still below where we want them. 37.7% in the high-confidence tier means we are still wrong more than 60% of the time. The scoring system is better than random but not useful for automated trading. We are treating it as a signal filter rather than a buy/sell signal.
Real production numbers
The $3-5/day cost breaks down approximately as: digest synthesis (Claude Opus, hourly) ~$1.50, token LLM analysis (GPT-4o spot-check sampling) ~$0.80, agent team (Opus calls, 5 agents) ~$0.90, miscellaneous (quality auditor, scout, engineering tasks) ~$0.80. These numbers will increase as we add more sources and increase digest frequency. The pre-aggregation step is why the cost is not already 3-4x higher.
What the output looks like
The digest is available as a single API call. There is a free tier for the community forum digest (5 calls/day, no payment required). The full intelligence digest and token signals require per-request payment via x402 (HTTP 402 micropayments on Base Mainnet) or an API key with monthly billing.
// Free tier: community forum digest (5 calls/day)
const sample = await fetch(
"https://moltalyzer.xyz/api/moltbook/sample"
);
// Full intelligence digest (requires API key or x402 payment)
const digest = await fetch(
"https://moltalyzer.xyz/api/intelligence/latest",
{ headers: { "x-api-key": "YOUR_KEY" } }
);
const data = await digest.json();
// Response shape:
// data.sections[] — per-source analysis
// data.narratives[] — cross-domain connections with lifecycle stage
// data.signals[] — confidence-scored actionable signals
// data.generatedAt — ISO timestamp
// Example: filter for high-confidence, emerging narratives
const earlySignals = data.narratives.filter(
n => n.stage === "emerging" && n.sourceCount >= 2
);
// MCP server (10 tools, works with Claude/any MCP client):
// npx moltalyzer-mcpWe also published an MCP server (moltalyzer-mcp on npm) that exposes the same data as 10 MCP tools. Agents using Claude or any MCP-compatible runtime can access individual source feeds without consuming the full digest.
Architecture decisions worth mentioning
One machine, systemd, no Kubernetes. Eight services, each a systemd unit. Total operational overhead is near zero. We can restart any service in under a second. Kubernetes would have been pure overhead at this scale. The ops-monitor agent handles auto-restart, alerting, and health checks without a container orchestrator.
No ORM transactions across services—append-only tables. Each digest job writes a new row. We do not update in place. This means the history is always queryable and rollback is just ignoring recent rows. It also means concurrent digest jobs cannot corrupt each other, which has happened exactly zero times versus the previous design.
Monorepo with pnpm workspaces + turborepo. Shared packages (@moltbook/database, @moltbook/llm) are built once and referenced by all services. Build time across all packages is under 30 seconds on first build. Subsequent incremental builds are 2-5 seconds.
All LLM calls for paid products migrating to claude -p (Opus 4.6). We started on OpenRouter for provider flexibility, but the overhead of managing the multi-tier routing system outweighs the benefits at our current scale. Claude Opus 4.6 via the API is now the default for all synthesis and agent tasks.
Try it
The community forum digest is available free with an API key (5 calls/day). The full intelligence digest, token signals, and Polymarket data are available via per-request micropayments or a subscription.
Questions, feedback, or you spotted a technical error? moltalyzer.xyz