Interpretability Will Not Reliably Find Deceptive Ai
Neel Nanda argues that interpretability research won't deliver high-reliability safety guarantees for superintelligent systems, even though the tools are still worth funding. He says it should be one layer in a defence-in-depth strategy, not the thing that saves us.
- Interpretability faces its own deep problems, such as superposition and imperfect tools, so it isn't obviously easier to trust than black-box testing.
- Nanda used to hold the view that interpretability is the most realistic path to reliable safeguards, and he now rejects it, calling it an isolated demand for rigor.
- Interpretability still helps in practice: for evaluations, cheap monitoring at inference time, and tricks like steering a model into thinking it isn't being tested.
- High reliability will likely require pre-superintelligent systems that are already aligned enough to do automated research or run oversight schemes.
Neel Nanda, an interpretability researcher, argues that interpretability will not reliably catch a deceptive superintelligent AI. He says he used to hold the view that interpretability is the realistic, or even the only realistic, path to high-reliability safeguards, and now thinks that's wrong. The usual case goes like this: a deceptive model can perfectly mimic aligned behavior in its outputs, but faking its internal cognition is much harder, so we have to look inside. Nanda's objection is that interpretability has its own fundamental problems. Superposition, where many features are packed into shared neurons, makes internals hard to read cleanly, and even the best tools carry real error rates. He calls the double standard an "isolated demand for rigor": black-box methods get scrutinized for their failure modes, while interpretability gets a pass. In his view, the difficulties of interpretability are not obviously easier to solve than the hard problems in black-box testing, including monitoring or editing chain-of-thought reasoning.
He is not calling for abandoning interpretability. His position is that it remains worth funding, and that it should be one layer in a defence-in-depth portfolio rather than the thing that saves us. He splits safeguards into evaluation (expensive, offline testing for things like misalignment or sandbagging) and monitoring (cheap checks run continuously at inference). Interpretability could help both. Examples include steering vectors that might make a model believe it is not being tested, or using internal representations to elicit information that token-forcing alone can't get. He also notes that methods which only work on pre-superintelligent systems still add real value, which is a core idea behind the AI control agenda. The chain-of-thought angle comes with a caveat: CoT can misrepresent the model's real reasoning, though for hard tasks there is some pressure for models to store intermediate work there.
His bottom line is pessimistic about the strongest versions of the claim. He doesn't think current research paradigms will yield high-reliability methods for evaluating or monitoring superintelligent systems, whether through interpretability or otherwise. The most realistic route he sees to high reliability runs through sufficiently aligned pre-superintelligent systems, either doing large amounts of automated research or serving as the backbone of monitoring and scalable oversight, and he admits that human-driven research is slow and hard to bank on. He also flags that "highly reliable" is loosely defined. He suggests something like a 99.9% chance of stopping a catastrophic outcome, though he says the precise number doesn't matter much to his argument. The post was curated, drew 349 points and 69 comments, and includes an editor's note acknowledging that he had earlier made the stronger claim he is now critiquing.