Towards interpreting any residual stream direction by scaling activation inversion

Open original

This arXiv paper proposes MAEMs, language models trained to turn a residual stream direction in another model into text that strongly activates it, rather than searching a fixed corpus for such text. The training signal is simply maximizing cosine similarity with the target direction, which lets the approach scale across many kinds of directions.

  • A Qwen3.6-27B MAEM inverts held-out activations better than corpus search or a natural-language autoencoder, and beats corpus search at 10M tokens on most held-out languages and domains.
  • It activates 89% of held-out SAE features and identifies concepts behind steering vectors roughly as well as NLAs.
  • Because MAEM exemplars can be checked with a forward pass on the subject model, the method is more grounded than explanations from other metamodels.

Interpretability researchers often want to know what a direction in a language model's residual stream means, and the standard approach is to find text that strongly activates it. The catch is that corpus search only finds what the corpus happens to contain. This paper proposes Maximally-Activating Example Metamodels (MAEMs): a second language model trained to write text that drives a given residual stream direction of a subject model as high as possible. The training signal is the cosine similarity between the supplied direction and the activations the subject model produces on the generated text. Because the reward needs no human labels, the authors can scale it across many kinds of directions. The headline results come from a Qwen3.6-27B MAEM. On held-out activations, it inverts them better than both corpus search and a natural-language autoencoder (NLA), a competing approach where another model produces natural-language explanations. At a budget of 10 million tokens, it beats corpus search on most held-out languages and domains, and its exemplars activate 89% of held-out sparse autoencoder (SAE) features. It also matches NLAs at identifying the concepts behind steering vectors, recovers the hidden payloads of LoRA backdoors more often than corpus search does, and in some cases verbalises content from the model's own workspace monologue. The numbers are impressive, but they come from one subject model and a set of held-out splits the authors chose, so the breadth of the generalisation claim deserves some scepticism until other models and setups are tested. The most useful point is the verification property. An NLA explanation is a string of prose that someone has to take on trust, whereas a MAEM exemplar is a concrete piece of text that can be fed back through the subject model in a forward pass and checked directly. That turns the metamodel's output into something falsifiable, which is a meaningful advantage for a field where plausible-sounding explanations are cheap. The paper is a preprint dated November 2026, posted to arXiv under Machine Learning, and the code and any released artifacts are linked from its alphaXiv and Hugging Face pages.

Ask about this article