@akshay_pachaar - LLM Inference: Prefill, Decode,
@akshay_pachaar on X A first-principles walk through what happens between your prompt and the streamed reply: tokenization, attention, and why the model does two very different jobs with two different bottlenecks. The piece argues that KV cache memory, not raw model intelligence, is what usually limits long-context speed.
- Generation is a loop that predicts one token at a time, and the first token is slower than the rest because prefill is compute-bound while decode is memory-bound.
- The KV cache avoids recomputing attention over the whole sequence, but at roughly 1 MB per token for a 13B model, a 4K context burns about 4 GB of VRAM.
- Quantization is the highest-leverage knob: a 7B model drops from 14 GB at FP16 to 3.5 GB at INT4, often with only a point of benchmark loss.
- DeepSeek's V4 previews reportedly cut the cache to about 10% of its predecessor's size at a one-million-token context, showing the field is redesigning attention around the cache.
Every LLM response is produced one token at a time, and the model runs two very different jobs to do it. First comes prefill: all your prompt tokens get processed in parallel, computing queries, keys, and values for the whole input at once. That is big matrix-by-matrix math, which GPUs are built for, so the bottleneck is raw compute, and the metric is time to first token. Then comes decode: the model appends one token at a time, computing only the new token's vectors and reusing cached keys and values for everything before it. The arithmetic per step is tiny, so the chip spends its time waiting on memory to load weights and cache. Same model, same GPU, opposite bottlenecks. That explains why the first token can take a few hundred milliseconds while the rest stream out at a steady clip, and why inter-token latency is what makes a model feel fast.
The KV cache is the piece that makes decode practical at all. Without it, each new token would require recomputing attention over the whole growing sequence. With it, you store each layer's keys and values once and reuse them. The speedup is roughly 5x or more on long generations, but the cache lives in GPU memory and grows with every token. The article puts a 13B model at about 1 MB per token, so a 4K context eats around 4 GB of VRAM just for the cache. That's why long contexts get slow and expensive: the cache runs out of room before the model runs out of capacity. The usual fixes are quantizing the cache to INT8 or INT4, sliding windows, grouped-query attention, and paging the cache the way an OS pages memory, which is the idea behind vLLM's PagedAttention.
The article also flags a shift in where research is heading. Rather than treating the cache as a fixed cost to trim, DeepSeek's V4 series, previewed in late 2025, redesigns attention so the cache is small from the start, using a hybrid of compressed sparse and dense attention. The reported figures at a one-million-token context are about 10% of the cache size and 27% of the per-token compute of the predecessor. Whatever you think of the specific architecture, it signals that the KV cache has become the constraint the field now designs models around. The piece also covers tokenization, embeddings, and quantization in more depth, and it's worth saving for the detail on those.