Click any tag below to further narrow down your results
+ long context
(2)
+ psycholinguistics
(1)
+ language models
(1)
+ activation inversion
(1)
+ maem
(1)
+ transformers
(1)
+ sparse attention
(1)
+ lsa
(1)
+ latent sparse attention
(1)
+ comprehension
(1)
+ information density
(1)
+ llm readability
(1)
+ eduslop
(1)
+ linguistic processing
(1)
+ agent state
(1)
Links
This arXiv paper claims an open 27B model can diagnose its own failures and edit its own weights, with no gradients and no training, and end up measurably more accurate. The authors frame it as "self-rewriting" and a possible step toward recursive self-improvement.
- The model's own edit raised its accuracy 6.6 points on problems it never saw during the edit.
- Across eight consecutive sessions it tested edits on practice problems and submitted only ones that worked, and accuracy only went up.
- Given a damaged copy of itself, it restored that copy to over 80% of original accuracy on unseen image tasks, without reading the original weights.
This arXiv paper proposes MAEMs, language models trained to turn a residual stream direction in another model into text that strongly activates it, rather than searching a fixed corpus for such text. The training signal is simply maximizing cosine similarity with the target direction, which lets the approach scale across many kinds of directions.
- A Qwen3.6-27B MAEM inverts held-out activations better than corpus search or a natural-language autoencoder, and beats corpus search at 10M tokens on most held-out languages and domains.
- It activates 89% of held-out SAE features and identifies concepts behind steering vectors roughly as well as NLAs.
- Because MAEM exemplars can be checked with a forward pass on the subject model, the method is more grounded than explanations from other metamodels.
This arXiv paper proposes Latent Sparse Attention (LSA), which clusters keys in a learned latent space so queries score cluster hierarchies rather than every key, aiming to make attention cheap at over 1M tokens. The authors report speedups over full and original sparse attention at 10M tokens while keeping accuracy close to the baseline.
- Full attention becomes too costly above 1M tokens, and existing sparse indexers either remain quadratic or lose accuracy at that scale.
- LSA's distilled variant (D-LSA) speeds up one attention layer at 10M tokens by 22.7x in prefill and 4.6x in decode over the original sparse attention, with similar accuracy.
- The native variant (N-LSA), trained only on next-token loss, matches full attention accuracy with a 1.6x per-layer training speedup at 128k tokens.
An arXiv preprint finds that frontier language models produce text full of features that slow human processing, like unusual syntax and rare word combinations. The authors argue this stems from models being optimized for information density in math, code, and reasoning rather than for being easy to follow.
- Compared with human writing, all nine tested frontier and near-frontier models use more inanimate subjects paired with animate verbs and more low-frequency n-grams.
- Most models also rely more on long dependencies, stacked nouns, and reduced object relatives, especially in conversational text.
- A proof-of-concept fine-tune under a length penalty pushed a model further away from human-like linguistic behavior, supporting the density explanation.
@monospodcast on X
A Google arXiv paper proposes replacing full conversation history in long AI agent sessions with a structured representation of current state, reportedly cutting token usage by 94%. The author, from the Monos podcast account, calls it potentially the best news for AI economics this year, at least for users.
- The method swaps the entire conversation history for a structured snapshot of the agent's current state.
- The paper reports a 94% reduction in token usage during long agent sessions.
- The author argues this matters most for users paying per-token costs, since long agent runs get expensive fast.
@not_ellington on X
A post from @not_ellington argues that sitting down to read and reread primary material beats the viral "educational" threads that rack up thousands of bookmarks. It links to an arXiv paper (2606.24937) as the kind of serious source worth that effort.
- The author calls viral 15-minute learning threads "eduslop," saying they get heavy bookmarking but teach less than the source material they summarize.
- The core claim is that the best way to learn is still to read something carefully and then reread it.
- The linked arXiv paper is offered as an example of the depth the author means, rather than a summary of a thread.