This gist provides a single-file, dependency-free implementation of a GPT-style transformer, complete with a custom autograd engine, training loop using Adam, and inference routine. It trains on a list of names, demonstrating both the core algorithm and a brief benchmark discussion for a GPU-based microgpt.cu variant.

• A complete GPT training pipeline—autodiff engine, transformer architecture, and Adam optimizer—fits in under 300 lines of pure Python with zero dependencies.
• The custom Value class implements backpropagation from scratch, proving you don't need PyTorch/TensorFlow to understand how gradients flow through a transformer.
• The model is genuinely tiny (16-dim embeddings, 4 heads, 16-token context, a few thousand params) yet demonstrates every core GPT component: embeddings, RMS-norm, scaled dot-product attention, residuals, and a feed-forward MLP.
• It's explicitly a stripped-down teaching tool, sacrificing batching and GPU speed to expose the raw mechanics of training a character-level name generator in 1,000 steps.

This article breaks down Andrej Karpathy’s zero-dependency, 243-line GPT implementation in plain Python. It explains how each part—tokenizer, autograd engine, embeddings, attention mechanism, residual connections, and MLP—mirrors a full-scale transformer on a tiny dataset of baby names.

• Karpathy's microGPT implements a full GPT—tokenizer, autograd engine, transformer, training loop—in just 243 lines of pure Python with zero external dependencies beyond os, math, random and argparse.
• A ~40-line custom autograd engine (Value class) replicates PyTorch's backward-pass mechanics via topological graph traversal.
• The toy model trains on baby names using a tiny architecture (16-dim embeddings, seq length 8, single layer, 4 attention heads) totaling about 4,000 parameters.
• The same core transformer math—embeddings, RMSNorm, attention, MLP—scales up unchanged to power trillion-parameter models like GPT-4.