@JayaGup10 - The Great Unbundling

Open original

@JayaGup10 on X The post argues that the all-purpose LLM is splitting apart: judgment, ranking, search, and verification are getting priced and built as separate cheap layers, with frontier models reserved for the hardest work. It introduces Jev from TypeSafe as an example of packaging one decision step as its own model call.

  • Agents make hundreds or thousands of model calls per task, so small per-step cost and latency differences add up to the whole product's economics.
  • A "decoder tax" is paid when a model generates a full sentence just so the app can turn it back into a yes/no, urgent/not-urgent, or which-tool decision.
  • Cheap enough judgment lets software check 100% of agent actions and customer interactions instead of sampling 1%, which opens product categories that aren't economical today.

The core claim is that the single general-purpose LLM, which has been the default answer to nearly every AI task for about three years, is about to split apart. Extraction, ranking, search, retrieval, judgment, browsing, verification, and hard reasoning all have very different cost and latency profiles, and different models are already better at different pieces. The author argues that once agents turn one objective into hundreds or thousands of model calls, small per-step cost differences compound into the economics of the product. Inference is now real cost of goods sold, smaller and open-weight models can absorb routine work, and better evals let applications tell when a cheap model is good enough. Frontier models getting better also helps, since a cheap layer can be used aggressively with a frontier model as the fallback. The architecture flips from "frontier by default, optimize later" to "cheap by default, frontier on exception." The piece uses a few concrete ideas to make this stick. The first is what it calls the "decoder tax": when an application only needs a decision like urgent or not, continue or stop, allowed or blocked, we often make a model generate a full text answer and then parse it back into that decision. A lot of agent inner-loop work could skip generation entirely. The author points to Jev, from TypeSafe, as a product that packages judgment at the model layer, returning structured decisions and probabilities over bounded answer sets, at a speed and cost suited to agent loops. The second idea is about margins. The author cites reports that Anthropic runs gross margins above 80%, and argues that a simple classification or verification call currently gets frontier pricing just because it happens inside a bundled call. Unbundling asks what each capability would cost as its own market. If routing sends the cheap, predictable, high-volume work to rerankers, search providers, code, browser models, and small models, the frontier lab is left with the hardest, most ambiguous, long-horizon problems. That means fewer calls reach the frontier, but each one may need more test-time compute, so the threat is less about cheaper frontier tokens and more about the decomposition of the workload underneath. The practical consequence the author cares most about is that the application layer becomes an "intelligence compiler." Instead of picking one model for a whole task, the app breaks a human objective into cognitive operations, buys the cheapest sufficient capability for each, recombines the results, and escalates only when needed. Once a capability has a stable interface, providers can be benchmarked, swapped, and priced independently, which the author says forces every capability to justify its place rather than eliminating the margin pool. Cheap judgment also changes what products are possible. At a fraction of a cent and a few hundred milliseconds, you can put a judge after every agent action instead of spot-checking, run intelligence inside browser and voice loops that multi-second inference would be too slow for, and move from sampling 1% of a system to evaluating all of it, such as every support conversation, transaction, or contract clause. The author's point is that most decisions software could make are never made today purely because intelligence is too expensive to apply continuously. Whether this unbundling actually plays out at the scale described is an open question the piece doesn't try to settle, but the economic logic it lays out is specific enough to test against real product budgets.

Ask about this article