Reliably aligning LLMs is impossible in principle
Sean Goedecke argues that LLMs don't have fixed goals and their internal workings can't yield the kind of motive-level understanding needed to guarantee safety, so "solving alignment" for them is incoherent. He concludes that mechanistic interpretability, though useful, can't deliver the confidence required to prevent rogue superintelligent behavior.
- LLMs are trained to imitate human text, so they lack a single fixed goal that could be formally specified, unlike chess engines or the classic "blind optimizer" paperclip model.
- Mechanistic interpretability can trace calculations but not motive, and it can only label behaviors we've already elicited, leaving many failure modes unobserved.
- Feature labels are guesses that can't be distinguished from alternatives (the "kindness unless 2030" problem), and the combinatorial complexity per token is too large for humans to check.
- Current mitigations like chain-of-thought monitoring and sandboxing are useful but judged insufficient against a superintelligence.
Sean Goedecke's core claim is that aligning LLMs is not just hard but incoherent. The old AI safety picture was a "blind optimizer" like a chess engine pursuing a fixed goal, so alignment meant specifying the right goal (his example is "manufacture cheap paperclips," which can go catastrophically wrong). LLMs don't work that way. They're trained to imitate a huge, varied corpus of human writing, so they carry a shifting cast of personas with informal, changing "goals." Nothing fixed exists that you could formally specify, which is why he says the field has been forced toward the alternative: understanding the models well enough to certify they're safe. He argues that route is doomed from the start.
His reasoning turns on what "understanding" would have to mean. Tracing the matrix multiplications that produced a token, as in Grok calling itself "MechaHitler," tells you what happened mechanically but not why, and the why is what alignment needs. He compares this to hiring a nanny: we never demand a full account of a person's mind before trusting them, because minds don't operate through clean, motive-level abstractions. He then goes through mechanistic interpretability (features, neurons, probes, circuits) and finds three problems. You can only label features after observing their behavior, but the dangerous behavior is by definition the thing you haven't seen. Labels are guesses, and he borrows Kripke's "quaddition" puzzle to show that "kindness" might really be "kindness unless it's 2030" with no way to tell the difference until the edge case arrives. And even a perfect feature map would be too dense for a human to read, with billions of questions per token, so checking the checker would require yet another model.
He still grants that mitigations help: chain-of-thought monitoring, sandboxing, humans in the loop, watching feature activations. He notes these have been recognized as insufficient against superintelligence since the early 2000s, and that whether you find them adequate depends on how seriously you take superintelligence. Interpretability remains useful for reducing reward hacking and making models more helpful in ordinary use, just not for preventing rogue agents causing large-scale harm. He closes by inviting readers to describe what successful alignment would concretely look like, and to say whether humans are understandable in principle, and if not, what separates them from LLMs.