What thousands of conversations look like inside a language model | Operator_0x
The author builds an interactive atlas of Qwen3-1.7B's residual stream, tracking how about 1,000 question-and-answer examples across four languages and Python land in the model's internal vector space. The result is a view of language and topic structure as overlapping geometric sheets.
- The axes track language strongly, but topic structure persists after accounting for language, so the model encodes both what is discussed and how it is said.
- Code examples trace paths through partly shared regions with natural-language examples, suggesting a common geometric structure across very different text.
- The headline 17.6% figure is token-state variance under the Global PCA axes, not a measure of meaning or reasoning, and the author cautions that the sample is small, templated, and single-turn rather than real multi-turn conversation.
The author took frozen base Qwen3-1.7B weights, with no fine-tuning adapter active, and recorded the residual stream: the model's shared working vector at each token position. The dataset was 900 question-and-answer examples per natural language, built from 100 topics, three question styles, and three examples per style, across English, Chinese, German, and Croatian. Questions came from templates and translated catalogues, and Qwen wrote the answers itself. A separate set of 90 Python coding examples, with English prompts, was added for comparison. The author is clear that this is single-turn Q&A, not real multi-turn conversation, so the "thousands of conversations" framing in the title is looser than it sounds.
The main finding is that language is the dominant axis of variation. In the three-dimensional Global PCA projection, each conversation forms a thin sheet, and the sheets for different languages occupy partly overlapping regions, with the code examples tracing their own paths. The axes track language strongly, but topic structure remains after language is accounted for, so the model is encoding both what is being discussed and how it is being discussed. The author ties this to earlier work on shared versus language-specific geometry in multilingual models. The write-up also flags that these are associations within this one dataset and projection, not general claims about how the model works.
The numbers need careful reading. The headline 17.6% is the share of token-state variation captured under the Global PCA axes, but those axes were fitted on a different pool (embeddings plus conversation averages) where the first three components retain only about 3.16% of variance. So the percentage is not a simple measure of how much the three axes explain. Token states are also weighted by count, meaning longer answers add more points. And the author stresses that variance is a measure of geometric spread, not of meaning or reasoning ability. Useful as an exploratory picture, but the figures should not be taken as evidence about what the model understands. Full measurements are linked as CSV and JSON if you want to check the arithmetic yourself.