Andrej Karpathy Just Built an Entire GPT in 243 Lines of Python | by Sumit Pandey | in Towards Deep Learning

Open original

This article breaks down Andrej Karpathy’s zero-dependency, 243-line GPT implementation in plain Python. It explains how each part—tokenizer, autograd engine, embeddings, attention mechanism, residual connections, and MLP—mirrors a full-scale transformer on a tiny dataset of baby names.

  • Karpathy's microGPT implements a full GPT—tokenizer, autograd engine, transformer, training loop—in just 243 lines of pure Python with zero external dependencies beyond os, math, random and argparse.
  • A ~40-line custom autograd engine (Value class) replicates PyTorch's backward-pass mechanics via topological graph traversal.
  • The toy model trains on baby names using a tiny architecture (16-dim embeddings, seq length 8, single layer, 4 attention heads) totaling about 4,000 parameters.
  • The same core transformer math—embeddings, RMSNorm, attention, MLP—scales up unchanged to power trillion-parameter models like GPT-4.

On February 11, 2026, Andrej Karpathy released microGPT: a single Python file—243 lines long—that builds, trains and runs a GPT model from scratch with no external dependencies beyond os, math, random and argparse. Instead of hundreds of packages you pip-install before even seeing code, microGPT stands alone. It downloads a list of baby names, converts each character into a number using two special tokens (<BOS>, <EOS>) plus the sorted unique characters from the dataset, and then learns to predict the next character in a name. The end result is a toy GPT that generates new, realistic-sounding names. About 40 lines of code implement an entire automatic differentiation engine mirroring PyTorch’s autograd. A custom Value class wraps every number, tracks its children in the computation graph, and defines backward functions for addition, multiplication and other ops. When you call .backward(), the code builds a topological ordering of operations and propagates gradients back through that graph. This lets the model adjust its randomly initialized weight matrices—embeddings, attention projections and feed-forward networks—using gradient descent. The transformer itself lives in roughly 100 lines. Karpathy defines embedding matrices (wte for token, wpe for position), draws random Gaussian weights, then in each layer applies RMSNorm, projects inputs into queries, keys and values, computes scaled dot-product attention across four heads, and feeds the result through a small two-layer MLP. With a 16-dimensional embedding, sequence length of eight and a single layer, microGPT uses about 4,000 parameters. The same architecture underpins GPT-4, but at a trillion-parameter scale. This project strips away every extra file and dependency to reveal the core math in clear, runnable Python.

Ask about this article