There's no point at which turning your brain off will work
Short summary
Dan Luu argues that while LLMs have improved at code generation, relying on them without human oversight produces broken software. The real problem isn't LLM capability—it's that people declaring "programming is solved" are shipping products that don't work and fooling themselves about quality standards.
- LLMs fail consistently on out-of-distribution problems, nuanced requirements, and tasks with hidden constraints—exactly what real software projects contain. They perform well only on narrow, well-defined tasks.
- When developers outsource thinking to LLMs and walk away, the software has serious bugs: infinite loops, broken functionality, overfitting to tests, and logic errors that code review catches immediately.
- The gap between "works for me" personal projects and shipping commercial products matters. People claiming LLMs solve programming often confuse the former with the latter, or ship low-quality code while ignoring obvious defects.
Full summary
Dan Luu argues that relying on LLMs to do your thinking—what he calls "brain-off" development—only works as a temporary strategy. The core problem: if an LLM can eventually produce good software without human oversight, why would a company keep paying the human? At that point, they'll just run the LLM in a loop and lay off the employee. Luu acknowledges LLMs have gotten better at this kind of work, but he's skeptical it scales to real problems. Luke Burton's response pushes back usefully: the tasks where you can walk away and let an agent run are probably low-value work anyway. High-value work requires constant judgment calls—understanding why code works a certain way, knowing what customers depend on, catching architectural mistakes before they compound. Burton's team spent months getting agents to convert code to Bazel, even with supervision, because the task had too many hidden requirements and decision points.
The real limitation shows up when agents hit unfamiliar territory. Luu uses board games as a concrete example: a modern LLM will confidently give advice about Dominion or Lost Cities that sounds plausible but is often wrong in ways that mislead a new player. A human who's never played before but spent five hours reading about the game would play circles around the AI. This gap matters for coding too. When you encounter code whose behavior isn't obvious, or when you need to know whether customers rely on some legacy feature, the agent performs poorly compared to a reasonable person. You still need someone watching for these out-of-distribution moments.
Luu has also noticed that people who hand their thinking to LLMs completely produce broken software. He checked out a prominent programmer's GitHub after they claimed Claude solved all their problems across different fields—the examples either didn't work or worked badly. He found an AlphaZero-style board game AI that lost to a simple minimax bot that an LLM tweaked in a loop. The pattern repeats: people assume the agent will handle everything, don't supervise closely, and end up shipping things that fail in obvious ways to anyone actually using them. The software works on paper but breaks in practice.