First Steps Toward Automated AI Research - Recursive
Short summary
Recursive built an AI agent that autonomously proposes, implements, and validates research ideas over long horizons. On the NanoChat benchmark it cut validation loss by 0.0263 BPB versus the community best and rediscovered and combined techniques like hashed bigram/trigram embeddings, optimizer tweaks, and architecture modifications.
- Recursive's autonomous AI research agent beat the community's best NanoChat result (0.9109 vs 0.9372 BPB) while also cutting training time 1.3x, running the full propose-implement-validate loop without human input.
- Starting from a bare Transformer+AdamW baseline (1.059 BPB), the system independently reached 0.9344 BPB, still beating the human-tuned public best.
- Two separate runs converged on similar tricks (hashed bigram/trigram embeddings, squared-ReLU MLPs) but also diverged with different novel additions (token shifting, weight averaging, byte-level embeddings), showing genuine exploration rather than repeating a fixed recipe.
- The winning approach added 1-2 billion sparse hashed n-gram parameters to a 50M-parameter model with minimal speed cost, using per-layer distinct hash primes to reduce collisions.
Full summary
Recursive’s AI research system runs the full research loop on its own: it proposes ideas, implements them, runs experiments, checks for reward hacks, and picks the next trial. On three benchmarks—fixed-budget language-model training, small-model speed, and GPU kernel optimization—it matched or beat state-of-the-art. The team has open-sourced the code and data so anyone can inspect their findings.
On the NanoChat autoresearch test, the system started from the same seed as Andrej Karpathy’s repo and bested the community’s collective effort. After stripping out minor hacks from the previous top solution and evaluating it over ten seeds, that human-agent mix averaged 0.9372 bits per byte (BPB). Recursive’s system hit 0.9109 BPB, shaving 1.3× off the training time to reach Karpathy’s original overnight quality. Even when it began from a plain Transformer with AdamW (1.059 BPB), it fell to 0.9344 BPB—again outdoing the public best.
The key innovations weren’t single tricks but a mix of tweaks across architecture, optimizer behavior, weight-decay schedules, compiler flags, and memory. A standout was hashed n-gram tables in the attention-value path. Instead of just standard value embeddings, the model uses bigram and trigram hash tables with learned gates. Each layer employs different hash primes to reduce repeat collisions. That adds up to 1–2 billion sparse parameters on a 50 M-parameter model without a big speed hit.
A parallel run on the vanilla Transformer rediscovered hash tables and squared-ReLU MLPs but also introduced things like token shifting, weight averaging before evaluation, and byte-level feature embeddings. Those differences show the system isn’t just replaying the same sequence of improvements twice. The code snippets in the article detail how layers mix gated bigram and trigram lookups into the value stream and assign distinct hash functions per layer.