Click any tag below to further narrow down your results
Links
The author builds an interactive atlas of Qwen3-1.7B's residual stream, tracking how about 1,000 question-and-answer examples across four languages and Python land in the model's internal vector space. The result is a view of language and topic structure as overlapping geometric sheets.
- The axes track language strongly, but topic structure persists after accounting for language, so the model encodes both what is discussed and how it is said.
- Code examples trace paths through partly shared regions with natural-language examples, suggesting a common geometric structure across very different text.
- The headline 17.6% figure is token-state variance under the Global PCA axes, not a measure of meaning or reasoning, and the author cautions that the sample is small, templated, and single-turn rather than real multi-turn conversation.
+ qwen3 1.7b
+ residual stream
+ representation geometry
interpretability
+ llm
+ multilingual
+ embeddings
This arXiv paper proposes MAEMs, language models trained to turn a residual stream direction in another model into text that strongly activates it, rather than searching a fixed corpus for such text. The training signal is simply maximizing cosine similarity with the target direction, which lets the approach scale across many kinds of directions.
- A Qwen3.6-27B MAEM inverts held-out activations better than corpus search or a natural-language autoencoder, and beats corpus search at 10M tokens on most held-out languages and domains.
- It activates 89% of held-out SAE features and identifies concepts behind steering vectors roughly as well as NLAs.
- Because MAEM exemplars can be checked with a forward pass on the subject model, the method is more grounded than explanations from other metamodels.
MIT CSAIL researchers had an AI model inspect and patch the numerical weights of a small open Qwen model, raising its MATH-500 score from 7.9% to 48.1% by touching about 0.016% of them. The gain came mostly from removing a repetition problem, not from adding new math ability.
- The edit changed only 131,072 weights in layers 15 and 19, cutting responses that hit the token limit from 93% to 33%.
- Finding that patch was expensive: the strongest run took 20.3 hours, 80 Python runs and 50 evaluation requests, including a 4.7-hour outage wait.
- A control on Gemma 3 270M showed no math gain, which suggests the method repairs flaws rather than adding knowledge.
- Critics split between doubting the reasoning claims and warning that the same tools could speed up capability development.
Anthropic is opening applications for a paid, six-week to six-month research residency that brings physicists, mathematicians, founders and senior engineers into full-time AI safety work, with no ML background required. Applications for this round close November 22, 2026.
- Pay is $350,000 to $500,000 a year equivalent, and residents are mentored one-on-one by Anthropic researchers and given a dedicated compute budget on frontier models.
- The program is built on the Anthropic Fellows model; past fellows produced work on subliminal learning, the Petri auditing tool, and persona vectors.
- Research areas include scalable oversight, adversarial robustness, interpretability, model organisms of misalignment, AI welfare, and training infrastructure. Residents are preferred in San Francisco, though remote participation is possible for shorter placements.
+ anthropic residency
+ anthropic fellows program
+ ai safety
+ alignment
interpretability
+ careers
This is an index page listing every post Sean Goedecke has tagged "explainers," covering topics like LLMs, DeepSeek, Model Context Protocol, blockchain, diffusion models, and interpretability. It's a jumping-off point rather than an article with a single argument.
- The index is dominated by AI topics: how LLMs and ChatGPT work, DeepSeek's cost and reasoning work, MCP, and interpretability.
- Blockchain shows up several times, including a Stripe "Tempo" FAQ and beginner-level primers.
- A few posts fall outside AI and blockchain, such as how to read code, code review mistakes, and writing technical explanations.
Neel Nanda argues that interpretability research won't deliver high-reliability safety guarantees for superintelligent systems, even though the tools are still worth funding. He says it should be one layer in a defence-in-depth strategy, not the thing that saves us.
- Interpretability faces its own deep problems, such as superposition and imperfect tools, so it isn't obviously easier to trust than black-box testing.
- Nanda used to hold the view that interpretability is the most realistic path to reliable safeguards, and he now rejects it, calling it an isolated demand for rigor.
- Interpretability still helps in practice: for evaluations, cheap monitoring at inference time, and tricks like steering a model into thinking it isn't being tested.
- High reliability will likely require pre-superintelligent systems that are already aligned enough to do automated research or run oversight schemes.
The author pushed each of Qwen3-8B's 442,368 MLP neurons up and down and asked the model "Who are you?", then catalogued every reply. A small share of the replies claim to be some other person or thing, and the site lets you browse them all.
- Of 884,736 neuron-direction pushes, 34,220 replies said the model was someone or something other than an AI.
- Each push was judged by Claude Haiku, not a human, so the persona labels are unverified.
- Persona changes don't prove the neuron encodes that persona; researchers disagree on what these pushes reveal, and the site doesn't settle it.
OpenAI's new Astra model uses recurrent loops to add computational depth, which technically hides AI reasoning in unreadable neural vectors rather than interpretable text. The company argues this poses no new safety risk since the added depth is modest compared to simply building deeper models, but the underlying concern—that AIs could think dangerous thoughts we can't monitor—remains unresolved.
- Neuralese recurrence lets AI loop internal representations (vectors) instead of outputting readable chain-of-thought text, making it impossible to monitor what the AI is actually thinking at scale.
- OpenAI claims Astra's recurrent depth (roughly 240 layers) is only 2x GPT-4's depth, so it's no worse than building a deeper model traditionally—a distinction that matters for safety but not for capabilities.
- Looping the same layers repeatedly is much weaker than adding new real layers; the AI can't learn new knowledge patterns, just apply existing ones more times, limiting the appeal of extreme looping.
TypeSafe AI released Jev, a model that takes text input but outputs confidence scores and probability distributions instead of generated text. It's designed for classification tasks like spam detection and ranking, costs $0.042 per million input tokens with free output, and has already spawned open-source recreations and creative experiments.
- Jev returns structured probabilistic outputs (yes/no scores, rankings, category choices) rather than text, making it fast and cheap—you only pay for input tokens, not output.
- The model trades interpretability for simplicity: you get a floating-point score but no explanation of which signals drove the decision, raising serious concerns about hidden bias in high-stakes use cases like hiring.
- The community has already built unconventional projects with it (a chat model, a game player, a left-pad implementation) and created open-weight alternatives like Kev within days of release.
Researchers propose a metric called NLS depth to quantify how much unobservable reasoning AI models can perform internally, separate from chain-of-thought outputs. This matters because architectural changes could let models do complex reasoning without showing their work, making them harder to monitor.
- Current chain-of-thought monitoring works because models are trained on human text, but new architectures could bypass this by performing reasoning in opaque hidden states instead of readable tokens.
- NLS depth measures the longest computational path that doesn't pass through natural-language outputs (tokens initialized from pre-training), making it possible to assess monitorability risk before a model is even trained.
- The metric deliberately ignores some post-training choices (like RLHF or distillation) to focus on architectural differences, distinguishing between standard transformers and genuinely concerning designs like full-bandwidth transformers that can hide reasoning across entire context windows.
OpenAI's Astra model uses "opaque recurrence," a technique that makes AI reasoning harder to monitor by processing queries in loops rather than linear steps. Safety researchers worry this could scale into a problem where AI reasoning becomes completely invisible.
- Opaque recurrence reduces legible traces of AI reasoning, making chain-of-thought monitoring less effective than current methods
- Safety experts fear the technique could escalate into a "race to the bottom" where labs stop maintaining transparent reasoning altogether
- OpenAI says Astra's use is limited and they remain committed to chain-of-thought monitoring, but Anthropic and Google DeepMind are already exploring the same approach
The author compares DNA genomes and large language model weights as passive information sequences—scores that only become meaningful when processed by cellular machinery or inference engines. Both arise from massive search processes (evolution and gradient descent) that compress vast experiences into lossy representations, leaving much uninterpreted “junk.” This parallel explains why it’s hard to pinpoint where specific functions reside in either genome or model.
- DNA and LLM weights are both inert symbol strings that only produce meaning when run through an interpreter (ribosomes/enzymes vs. inference engines)
- Both are compressed, lossy residues of massive iterative search—4 billion years of natural selection vs. gradient descent over trillions of examples—not literal copies of the experiences that shaped them
- Neither system stores discrete facts or instructions at a single location (no "gene for a hand," no weight for "Paris is the capital of France"); competence is distributed across the whole structure, which is why pinpointing function in either genome or model is so hard
+ dna llm analogy
+ distributed function
+ lossy compression
interpretability
+ evolution
+ emergence
Pingkit is a toolkit designed for training reproducible, capacity-aware models using transformer activations. It offers features for extracting embeddings, training neural architectures, and creating custom probes tailored to specific research needs. The toolkit is integrated with Hugging Face models and provides various utilities for data processing and model training.
- Pingkit extracts embeddings directly from transformer activations for downstream probing/analysis rather than relying on final-layer outputs alone.
- It's built to train "capacity-aware" probes, letting researchers control model complexity to avoid overstating what information is linearly/nonlinearly decodable from activations.
- Integrates with Hugging Face models, so it can plug into existing pretrained transformer workflows without custom extraction code.
- Emphasizes reproducibility, aiming to standardize probing experiments that are often ad hoc and hard to replicate across papers.