microgpt
This gist provides a single-file, dependency-free implementation of a GPT-style transformer, complete with a custom autograd engine, training loop using Adam, and inference routine. It trains on a list of names, demonstrating both the core algorithm and a brief benchmark discussion for a GPU-based microgpt.cu variant.
- A complete GPT training pipeline—autodiff engine, transformer architecture, and Adam optimizer—fits in under 300 lines of pure Python with zero dependencies.
- The custom Value class implements backpropagation from scratch, proving you don't need PyTorch/TensorFlow to understand how gradients flow through a transformer.
- The model is genuinely tiny (16-dim embeddings, 4 heads, 16-token context, a few thousand params) yet demonstrates every core GPT component: embeddings, RMS-norm, scaled dot-product attention, residuals, and a feed-forward MLP.
- It's explicitly a stripped-down teaching tool, sacrificing batching and GPU speed to expose the raw mechanics of training a character-level name generator in 1,000 steps.
Andrej Karpathy’s gist lays out a complete GPT training loop in under 300 lines of pure Python, no dependencies. It begins by fetching a simple text dataset—here, a list of names—and builds a character-level tokenizer that assigns each unique character an integer ID plus one special “beginning of sequence” token. From there he defines a tiny autodiff engine (class Value) that tracks computations and gradients through addition, multiplication, exponentiation, ReLU and so on. That engine underpins both the forward pass and back-propagation without relying on PyTorch or TensorFlow.
Next comes the model itself: a one-layer transformer with 16-dim embeddings, 4 attention heads, 16-token context window, and roughly a few thousand parameters (embeddings, query/key/value/projection matrices, feed-forward weights). Position embeddings and token embeddings get summed, then flow through an RMS-norm, a multi-head self-attention block with dot-product scaled by √d, plus residual connections, followed by a two-layer feed-forward network with ReLU. Logits over the next character are produced by a final linear “lm_head” layer.
Training uses plain stochastic gradient descent via the Adam update rule (learning rate 0.01, β1=0.85, β2=0.99, ε=1e-8). At each step, a single name sequence (padded with BOS tokens) is passed through the model, the cross-entropy loss is averaged over up to 16 positions, then back-propagated. Parameter updates happen immediately using the moment estimates stored in two Python lists. A thousand training steps wind up teaching the network how to predict the next character in simple name strings.
Everything outside of the algorithm—batching, fast GPU kernels, optimized matrix math—is stripped away. What remains is a didactic, end-to-end blueprint of how a GPT-style transformer works under the hood: from raw token IDs through self-attention, MLP layers, loss computation and gradient descent, all expressed in straightforward Python.