LLMs are terrible at designing AI systems
The author argues that coding agents, asked to build software that itself calls LLMs, reliably reach for brittle non-LLM approaches like regex or symbolic solvers. He offers three possible explanations and leans toward the idea that current agents are simply poor at software design.
- In one vibe-coded memory-contradiction tool, the agent built a complex symbolic logic system that couldn't work, even though it later admitted the underlying problem was a 130-year-old unsolved one; a single LLM call would have been the right tool.
- The author's own Autodeck agents kept categorizing flashcards with regex and keyword search instead of making another LLM call.
- Three candidate explanations are weighed: training code predates LLMs (ruled out, since models don't default to jQuery-era patterns either), RL environments rarely expose LLM API access, and agents are just bad at software design, which the author now finds most convincing.
LLMs are surprisingly bad at designing software that itself uses LLMs, even though they handle ordinary programming well. Sean Goedecke's central example is a vibe-coded memory tool meant to flag when one stored memory contradicts another. The obvious approach is to have an LLM scan for contradictions, either in batches or pairwise, which is now easy. But when asked for the state of the art, the agent (Claude, in this case) built an elaborate symbolic pipeline that converted memories into logical atoms and compared them with a solver. That cannot work, since no analytic method reliably decides whether two natural-language sentences express the same proposition. When the user finally asked why nothing was working, Claude immediately acknowledged the problem was a known 130-year-old dead end, yet it had spent weeks iterating on it. Goedecke reports a similar pattern in his own project, Autodeck, where agents kept categorizing flashcards with regex and keyword search instead of making another LLM call.
He offers three explanations. The first is that most training code predates LLMs, so models default to older tools. He rejects this because models don't default to jQuery or Backbone either, even though those are also old, so it can't be the whole story. The second is that reinforcement learning environments rarely include API access for models to call other models, so RL training may actively discourage that style of code. He admits he doesn't know this for certain, but notes that it would be expensive and risky to set up, and it would explain why LLMs are good at orchestrating subagents, which is likely something they've been directly trained on. His preferred explanation is that current agents are simply weak at software design, choosing doomed approaches like complex consensus algorithms or synchronous checks inside event-driven systems. He says he used to attribute these failures to alignment but now thinks they may just be ordinary design errors.
The practical takeaway, in his view, is that you can't judge an LLM by human standards. A human engineer capable of building a sophisticated symbolic solver would know it wouldn't work, so their suggestion would signal a good reason. An LLM can be both brilliant and oddly foolish on the same task. Using these tools well means predicting in advance where they'll succeed or fail, and he argues that without that prediction, you'll waste substantial time and tokens chasing hopeless approaches.