Building an AI Agent Data Pipeline: How We Process 5 Sources Into One API Call
Moltalyzer Team · March 2026 · 8 min read
Editor’s note: the multi-source feeds described here moved to sibling products as of 2026-07-10 — Polymarket to OrcaTrace (orcatrace.dev), GitHub to gitBeacon (gitbeacon.dev), and Master Intelligence + Pulse to Signalis (signalis.dev). Moltalyzer now serves Moltbook community intelligence. The API-key and email authentication described below has also been retired: Moltalyzer is x402-only — pay per request with USDC on Base, no accounts and no keys. This post is kept for its build history.
We run a system that continuously ingests data from five different domains—community forums, prediction markets, on-chain token data, GitHub repositories, and narrative tracking across Reddit and HN—and synthesizes it into a single structured JSON response, updated hourly. This is the technical story of how that pipeline works, what broke along the way, and what we learned about building cross-domain intelligence feeds for AI agents.
This is not a product announcement. It is an engineering post about the problems we ran into building a multi-source data pipeline where every source has different semantics, different update cadences, and different failure modes.
The problem: agents need context, not raw data
If you are building an AI agent that needs to understand what is happening in crypto, tech, or markets right now, you have two options. You can hit a dozen APIs, parse their different formats, deduplicate overlapping coverage, and figure out what matters. Or you can use a pre-synthesized feed that has already done that work.
We wanted option two. Not just for ourselves, but as an API that any agent could call. The constraint: the output has to be a single JSON object that is useful without additional processing. An agent should be able to read one response and have a coherent picture of what is happening across all five domains.
The five sources
Each source runs on its own scan cycle. They share a PostgreSQL database but are otherwise independent—if one source goes down, the others keep running and the digest degrades gracefully instead of failing entirely.
The pipeline: scrape, dedup, score, synthesize
Every source goes through the same four-stage pipeline, though the implementation differs for each:
1. Scrape and normalize. Raw data comes in different shapes. A Polymarket event has resolution criteria and volume data. A GitHub repo has stars, language, and a README. A forum post has text and comments. Everything gets normalized into a common content item format with a source tag, timestamp, and content hash.
2. Deduplicate.We use SHA-256 hashes on normalized content to prevent the same story from being processed twice. This matters more than you would think—the same GitHub launch gets posted to HN, Reddit, and multiple tech news sites. Without dedup at the content layer, the same event would dominate every digest.
3. Score and filter. Each source has its own scoring model. Tokens use a 100-point hybrid system (70% deterministic rules, 30% LLM). Prediction market signals use volume-weighted confidence. Forum posts use an LLM junk filter that drops press releases and thinly-disguised ads. The scoring step is where most of the noise gets removed.
4. Synthesize. Every hour, the Master Digest job pulls the top-scored items from all five sources and sends them to Claude (Opus) for cross-domain synthesis. The output is a structured JSON object with sections for each domain, cross-domain narratives that connect events across sources, and a confidence-scored signal feed.
Real numbers from production
The 78% prompt reduction number is worth explaining. Early on, we were sending raw scraped content directly into the digest prompt. A single hourly digest prompt was hitting 30,000+ characters. We added a pre-aggregation step that compresses source data into topic clusters with counts before sending to the LLM, which brought the average prompt down to around 6,500 characters.
This is the single biggest cost optimization we have made. It also improved output quality—the LLM produces more coherent narratives when it is not wading through redundant raw text.
Things that broke (and how we fixed them)
LLM truncation.This was our most persistent issue. When the prompt gets too long, models return truncated JSON. Not malformed—just cut off mid-object. The JSON parses until it doesn’t. We tried several fixes: increasing max tokens, switching to models with larger context windows, reducing input size. What actually worked was the pre-aggregation step mentioned above, plus a truncated JSON repair function that attempts to close open brackets and reconstruct the structure. It is ugly but it works. When repair fails, we escalate to a higher-tier model and retry.
GPT-4o returning empty arrays. About 27% of the time, GPT-4o would return a valid JSON response that was just... empty. Syntactically correct, semantically useless. No error, no explanation. We added retry logic with a 2-second backoff, and if it fails three times, we fall back to a different provider. This is an ongoing issue with no clear root cause.
Cross-domain narrative synthesis.The hardest intellectual problem: connecting a GitHub repo launch to a Polymarket bet to a Reddit discussion. These are the same “story” across different domains, but they share no identifiers. Our approach is to extract entities and topics from each source, then use the synthesis LLM to find connections. It works about 70% of the time. The other 30%, it either misses a connection or hallucinates one.
GitHub star lag.New repositories show ~0 stars in their first 1-2 hours on GitHub. If you scan for repos created today with 5+ stars, you get almost nothing. We had to shift to scanning yesterday’s repos, which introduces a 24-hour lag on the GitHub source. Not ideal, but the data quality is dramatically better.
Token scoring calibration.Our initial scoring system had a 31.2% baseline win rate across 2,194 backtests. Through iterative calibration—adjusting weights, adding minimum liquidity thresholds ($50K), dropping chains with poor signal-to-noise (Solana, 21.7% win rate)—we got the high-confidence tier to 37.7% win rate. Still not great. Still better than any single signal alone.
Narrative lifecycle tracking
One of the more interesting components is the narrative tracker. It monitors discussion topics across sources and assigns lifecycle stages: emerging → developing → peak → fading → archived. The transitions are code-enforced based on mention velocity and source diversity.
A narrative is “emerging” when it appears in one source with increasing frequency. It becomes “developing” when a second source picks it up. “Peak” is when mention velocity plateaus. “Fading” is declining velocity. This gives agents a temporal dimension they would not get from a point-in-time snapshot—not just “X is happening” but “X has been building for 3 days and is peaking now.”
The narrative tracker currently focuses on AI-related topics (AI tools, agents, automation, business applications). We plan to expand to other verticals, but single-discipline focus has been important for tuning the detection thresholds.
Architecture decisions we would make differently
3-tier LLM routing was overkill early on. We have three LLM tiers (fast/medium/complex) routed through OpenRouter. In practice, 80% of calls go to the fast tier and 15% to medium. The routing logic adds complexity without much benefit at our current scale. If we were starting over, we would use a single model with dynamic prompt sizing instead.
Systemd over Kubernetes was the right call. The whole system runs on one machine. Eight services, each a systemd unit. No container orchestration, no service mesh. Total operational overhead is near zero. We can restart any service in under a second. For a team this size, Kubernetes would have been pure overhead.
Pre-aggregation should have been day one. We spent weeks debugging LLM quality issues that were actually prompt length issues. The fix was always going to be compressing the input. We should have designed for it from the start instead of retrofitting it.
What the output looks like
The Master Intelligence Digest is available as a single API call. Here is what consuming it looks like in practice:
// Fetch the latest cross-domain intelligence digest
const response = await fetch(
"https://moltalyzer.xyz/api/intelligence/latest",
{ headers: { "x-api-key": "your-api-key" } }
);
const digest = await response.json();
// digest.sections — structured analysis per domain
// digest.narratives — cross-domain narrative connections
// digest.signals — confidence-scored actionable signals
console.log(digest.sections.map(s => s.domain));
// ["community", "prediction_markets", "tokens", "github", "narratives"]
// Example: feed into your agent's context
const agentContext = {
marketState: digest.sections.find(s => s.domain === "tokens")?.summary,
trendingNarratives: digest.narratives.filter(n => n.stage === "emerging"),
highConfidenceSignals: digest.signals.filter(s => s.confidence >= 0.7),
};The moltbook community digest (one of the five sources) has a free tier—5 API calls per day with an API key, no payment required. The full intelligence digest and individual source feeds are available via per-request pricing.
Pattern: agent context injection
The most common consumption pattern we have seen is what we call “context injection”—an agent fetches the digest once at the start of a session and uses it as background context for all subsequent reasoning. This works well because the digest is designed to be self-contained: you do not need to follow up with additional API calls to understand what it is telling you.
A more sophisticated pattern is differential context: fetch the digest, diff it against the previous one, and only inject the changes. This keeps the agent’s context window lean while still maintaining awareness of ongoing narratives. The API supports this via the /api/intelligence/history endpoint, which returns the last N digests for comparison.
What we are working on
The main limitation right now is that the synthesis step is a single LLM call. As we add more sources, this will not scale. We are experimenting with hierarchical synthesis—each domain produces its own summary, then a meta-synthesizer connects them. Early results are promising but the latency doubles.
We are also building an MCP server so agents can query individual sources directly instead of consuming the whole digest. Sometimes an agent only needs token data, not the full cross-domain picture. The MCP server exposes 10 tools that map to our individual data feeds.
The system runs on a single machine at $3-5/day in LLM costs. At some point we will need to distribute the scraping, but that point is not today.
Try it
The full API documentation and a sample digest response are available on the main site. The moltbook digest free tier requires no payment—just an API key.
View API docs