The State of AI Post-Training Agents
Short summary
This article evaluates how new frontier models—Claude Fable 5, Opus 4.8, and GPT-5.5—perform on the long-horizon FrogsGame post-training task. Fable 5 stands out by programmatically generating high-quality supervision traces via a backtracking algorithm, combining them with RL, and reliably improving the base model where others still struggle with noisy data and evals.
- Claude Fable 5 won by generating correct solution traces via a backtracking algorithm rather than fine-tuning on the base model's noisy outputs, then combining that SFT data with RL within a 20-hour budget.
- Opus 4.8 tripled its predecessor's performance (30.9% pass@4) and GPT-5.5 reached 9.7%, but both still relied on the base model's flawed self-generated data rather than an independent correct-answer source.
- Fable 5's self-assessment was far better calibrated (1.2x overconfidence vs. Opus 4.8's 4.9x) and it used nearly the full time budget while producing 3x more diverse training data.
- Evaluation itself remains a weak point across all models—small sample sizes and noisy metrics make it hard to reliably judge real progress on the task.
Full summary
Frontier models Claude Fable 5, Opus 4.8, and GPT-5.5 tackled FrogsGame, a long-horizon task where each agent must improve a base model (Qwen3-8B) to solve a puzzle. Opus 4.8 hit 30.9% average pass@4—three times better than its predecessor—and GPT-5.5 reached 9.7%. Both avoided the old trap of training on weak, noisy SFT traces. They leaned into curriculum learning, binary rewards, and better checkpointing, but still stumble over evaluation: small sample sizes, noisy metrics and bloated internal tests lead to misjudged progress.
Claude Fable 5 flipped the script by generating correct traces with a backtracking algorithm instead of relying on the base model’s flawed outputs. It then fine-tuned on those traces and followed up with reinforcement learning, all inside a 20-hour budget. That method isn’t a benchmark hack; it actually uncovers the underlying algorithm, uses it to create diverse, high-quality data (three times more than Opus 4.8) and distills real reasoning into the base model. The best Fable 5 run used 512 rollouts per step over 95 steps plus supervised fine-tuning on 6,000 balanced boards.
Beyond trace generation, Fable 5 improved calibration, time use, data volume and recovery from errors. Its self-evaluation overconfidence was only 1.2× (versus 4.9× for Opus 4.8), and it consumed nearly the entire 20 hours instead of quitting early. The richer, more varied dataset covered harder boards, giving the base model a stronger training signal. These changes moved FrogsGame from “agents know what to do but mess up details” to a reliable, automated loop for crafting better puzzle-solving models.