Latent Sparse Attention: Towards Subquadratic Ultra-Long Context Attention

Open original

This arXiv paper proposes Latent Sparse Attention (LSA), which clusters keys in a learned latent space so queries score cluster hierarchies rather than every key, aiming to make attention cheap at over 1M tokens. The authors report speedups over full and original sparse attention at 10M tokens while keeping accuracy close to the baseline.

  • Full attention becomes too costly above 1M tokens, and existing sparse indexers either remain quadratic or lose accuracy at that scale.
  • LSA's distilled variant (D-LSA) speeds up one attention layer at 10M tokens by 22.7x in prefill and 4.6x in decode over the original sparse attention, with similar accuracy.
  • The native variant (N-LSA), trained only on next-token loss, matches full attention accuracy with a 1.6x per-layer training speedup at 128k tokens.

Long-horizon agentic work, like open-ended scientific research, pushes models toward context windows above a million tokens, and at that scale full attention becomes too expensive to run. Sparse attention, where a separate indexer picks a subset of keys for each query, is the usual fix. The authors argue that existing indexers either still scale quadratically with context length or lose accuracy once the context gets very large. Their proposal, Latent Sparse Attention (LSA), keeps the indexing cheap by clustering keys hierarchically in a learned latent space. Queries then score the cluster hierarchy rather than every individual key, which is what makes prefill, decoding, and training faster at long context. The paper offers two variants. Distilled-LSA (D-LSA) is meant to be bolted onto an existing pretrained model through distillation, while Native-LSA (N-LSA) is trained from scratch using only the next-token loss. The headline numbers come from a 10-million-token setting. On GLM-5.3-FLASH, D-LSA is reported to speed up a single attention layer by 22.7x in prefill and 4.6x in decode compared with the model's original sparse attention, and the comparison against full attention on QWEN-3.8-27B is larger still, at 72.4x for prefill and 42.5x for decode. The authors say accuracy stays similar across a range of tasks. N-LSA is the more modest claim: it matches full attention's accuracy and trains 1.6x faster per layer at 128k tokens. Note that the speedup baselines differ between the two models, so the numbers aren't a single apples-to-apples comparison. What to check before trusting this: the abstract gives per-layer speedups and says accuracy is "similar" without naming the benchmarks or the size of any gap, so the full results tables matter more than the summary. Also worth confirming is whether the 10M-token figures come from measured runs or extrapolation, and whether end-to-end model speedups hold up once you account for layers that aren't accelerated. The paper is a 30 October 2026 arXiv submission, and the "experimental" HTML version plus the model names suggest it's an early-stage report. For a read-it-later queue, it's worth saving for the methods section and the ablations, and skimming the abstract alone won't tell you whether the accuracy claims survive scrutiny.

Ask about this article