Part 1/3 Owning Intelligence Series: How Coinbase Built, Evaluated, and Post-Trained a Fraud Agent

Open original

Coinbase built an LLM-powered fraud detection agent for its Onramp crypto payment service that sits downstream of existing ML models and rules, reviewing transaction sequences for patterns that emerge across multiple purchases. In live testing, the agent reduced fraudulent transactions by 30% and fraud value by 22%, with projected fraud prevention savings roughly three times the inference cost.

  • The agent catches fraud patterns invisible in single transactions by reviewing structured context, historical summaries, and transaction sequences together—something the existing ML model and rules engine miss.
  • Selective routing of transactions to the LLM balances fraud detection against latency and cost, with the agent only overriding decisions to block (never to allow) what the upstream checks flagged.
  • Newer model versions don't automatically improve domain-specific performance: Part 2 found that newer Opus, Sonnet, and GPT releases all scored lower on recall and F1 for fraud detection, motivating Coinbase to post-train an open-weight model instead.

Coinbase built a fraud detection system for their Onramp product (which lets people buy crypto in partner apps without a full account) by layering an LLM agent on top of their existing ML model and rules engine. The key insight was that some fraud patterns only show up across multiple transactions rather than within a single purchase—a single transaction might look normal, but the sequence reveals something suspicious. They routed transactions selectively to the LLM to balance detection, latency, and cost, giving it structured context including transaction details, historical activity summaries, and domain-specific guidance. An online A/B test showed the agent caught patterns the first layer missed, resulting in 30% fewer fraudulent transactions and 22% less fraud value. The fraud prevention savings were roughly three times the inference costs. The system was designed for visibility and reliability. Every serving response included the agent's decision, risk level, model used, latency, retry count, and token usage—separating detection behavior from serving behavior so changes in outcomes could be traced to specific causes rather than treated as mysterious shifts. The agent could only add an extra block to transactions, never override upstream blocks from the ML model or rules, keeping its authority narrow and its role complementary rather than replacing existing safeguards. This success raised a harder question: does the improvement hold when you swap in a different model? Coinbase doesn't control the weights of closed-source models, so version changes can shift risk judgments even when everything else stays the same. General-purpose benchmarks don't tell you whether a newer model actually helps or hurts at fraud detection. That gap motivated Parts 2 and 3 of the series—building domain-specific evaluation to compare models, then post-training an open-weight model on their proprietary fraud data to own the entire system.

Ask about this article