This article traces the path from RNNs to transformers, explaining why attention-based, non-recurrent architectures replaced older models. It then breaks down encoder vs decoder designs and shows how GPT’s decoder-only approach and tokenization power today’s large language models.

• Transformers replaced RNNs by dropping recurrence entirely, using self-attention to compute all token relationships in parallel and eliminating both the sequential bottleneck and vanishing/exploding gradient issues.
• The original transformer architecture split into two lineages: encoder-only (BERT) for contextual understanding, used in Google Search ranking, and decoder-only (GPT) for text generation.
• GPT's scale jumped dramatically across versions—GPT-2 had 10x GPT-1's parameters/data, GPT-3 scaled another 100x—and GPT-3 showed strong zero-shot/few-shot performance without needing task-specific fine-tuning.

This gist provides a single-file, dependency-free implementation of a GPT-style transformer, complete with a custom autograd engine, training loop using Adam, and inference routine. It trains on a list of names, demonstrating both the core algorithm and a brief benchmark discussion for a GPU-based microgpt.cu variant.

• A complete GPT training pipeline—autodiff engine, transformer architecture, and Adam optimizer—fits in under 300 lines of pure Python with zero dependencies.
• The custom Value class implements backpropagation from scratch, proving you don't need PyTorch/TensorFlow to understand how gradients flow through a transformer.
• The model is genuinely tiny (16-dim embeddings, 4 heads, 16-token context, a few thousand params) yet demonstrates every core GPT component: embeddings, RMS-norm, scaled dot-product attention, residuals, and a feed-forward MLP.
• It's explicitly a stripped-down teaching tool, sacrificing batching and GPU speed to expose the raw mechanics of training a character-level name generator in 1,000 steps.