Click any tag below to further narrow down your results
Links
This article traces the path from RNNs to transformers, explaining why attention-based, non-recurrent architectures replaced older models. It then breaks down encoder vs decoder designs and shows how GPT’s decoder-only approach and tokenization power today’s large language models.
- Transformers replaced RNNs by dropping recurrence entirely, using self-attention to compute all token relationships in parallel and eliminating both the sequential bottleneck and vanishing/exploding gradient issues.
- The original transformer architecture split into two lineages: encoder-only (BERT) for contextual understanding, used in Google Search ranking, and decoder-only (GPT) for text generation.
- GPT's scale jumped dramatically across versions—GPT-2 had 10x GPT-1's parameters/data, GPT-3 scaled another 100x—and GPT-3 showed strong zero-shot/few-shot performance without needing task-specific fine-tuning.
This article breaks down the core concepts behind LLMs—from next-token prediction training to tokens, vectors and attention layers—to show how they generate text. It also covers context windows, parameters and why model scale affects performance.
- LLMs work purely through next-token prediction learned by hiding and guessing words billions of times during training, not through actual understanding of letters or math.
- Context windows now range wildly, from 200K tokens (~150K words) up to Llama 4 Scout's 10 million tokens, but bigger windows don't fix factual errors or logical gaps.
- Because models process text as tokens/vectors rather than raw letters, they inherently struggle with tasks like counting letters or doing arithmetic.