This article traces the path from RNNs to transformers, explaining why attention-based, non-recurrent architectures replaced older models. It then breaks down encoder vs decoder designs and shows how GPT’s decoder-only approach and tokenization power today’s large language models.

• Transformers replaced RNNs by dropping recurrence entirely, using self-attention to compute all token relationships in parallel and eliminating both the sequential bottleneck and vanishing/exploding gradient issues.
• The original transformer architecture split into two lineages: encoder-only (BERT) for contextual understanding, used in Google Search ranking, and decoder-only (GPT) for text generation.
• GPT's scale jumped dramatically across versions—GPT-2 had 10x GPT-1's parameters/data, GPT-3 scaled another 100x—and GPT-3 showed strong zero-shot/few-shot performance without needing task-specific fine-tuning.

The author frames tokenizer design as an integer linear program, relaxes it to a continuous LP, and uses cutting planes to close the gap between fractional and integral solutions. They automate cut discovery with Codex, apply cycle constraints on overlapping token edges, and report provably optimal tokenizers on small pretokenized datasets.

• BPE, despite being greedy, actually gets within 1% of the true optimal tokenizer, as revealed by casting tokenization as an ILP with a provable lower bound.
• The LP relaxation plus rounding gives a feasible upper bound, and cutting planes (constraints violated by fractional solutions but satisfied by all integral ones) tighten the gap between the two.
• Codex was used to automatically discover these cutting planes by brute-forcing small word pairs/triplets and solving auxiliary LPs to find maximally violated constraints, rather than deriving them by hand.
• Results are only provably optimal under a fixed pretokenization scheme, since dropping that assumption would make the LP intractably large.

This article breaks down how an LLM turns your prompt into streamed tokens, covering tokenization, embeddings, transformer attention, and the two-phase pipeline of compute-bound prefill and memory-bound decode. It explains KV caching, quantization, and metrics like Time to First Token and Inter-Token Latency to show why inference speed depends on both compute and memory.

• LLM inference splits into compute-bound prefill (TTFT) and memory-bound decode (ITL), which explains why the two phases have fundamentally different performance bottlenecks.
• KV caching avoids quadratic recomputation but costs real memory—about 1 MB per token for a 13B model, so a 4,000-token context burns 4 GB of VRAM.
• Techniques like INT8/INT4 cache quantization, token eviction, and paged cache management exist specifically to tame this KV cache memory growth.
• DeepSeek's V4 cuts KV cache size by 90% at a million-token context by combining sparse and dense compressed attention.

A16z outlines 17 key developments expected in the crypto landscape by 2026, focusing on innovations in stablecoins, tokenization of real-world assets, and the transformation of financial systems through blockchain technology. The article emphasizes the role of stablecoins in modernizing payment infrastructures and the potential for personalized wealth management accessible to a broader audience.

• Stablecoins are positioned to modernize payment infrastructure by replacing slow, costly legacy rails with faster settlement
• Tokenization of real-world assets is expected to accelerate, expanding blockchain use beyond crypto-native assets
• Personalized wealth management tools built on blockchain are expected to become accessible to a much broader, non-wealthy audience