We had to build new evals for Fable

Open original

Hex built a suite of analytical evals to test data-analysis models and found Claude Fable 5 outperforms its Opus 4.x predecessors by 10–15%, nailing both semantically modeled and raw-data tasks with fewer mistakes. They’ve also designed a tougher “Frontier” benchmark for long-horizon, open-ended scenarios, where Fable 5’s careful assumptions and cross-checks boost its pass rate to around 58%.

  • Claude Fable 5 beats Opus 4.7 by 10-15 points on Hex's core benchmarks, scoring 93%+ on Analytical Hard/Semantically Modeled tests and 65% on Semantically Unmodeled tasks, versus prior Opus versions' single-digit gains
  • Fable's advantage comes from following a "golden workflow" (starting in the semantic layer, cross-checking raw SQL) and transparently stating assumptions, which lets it catch errors like a cents-for-dollars mistake that Opus misses
  • On Hex's new "Frontier" benchmark for long-horizon, open-ended tasks, Fable at Max Effort hits 58% pass rate, notably outperforming other setups

Anthropic’s Claude Fable 5 breaks new ground in analytical reasoning, outpacing its predecessor Opus 4.7 by 10–15 percentage points on Hex’s core benchmarks. On Analytical Hard and Semantically Modeled tests, it scores over 93%, and it hits 65% on Semantically Unmodeled tasks—numbers that approach the practical ceiling for single-turn evaluations. Earlier Opus releases only moved those metrics in single-digit increments, sometimes even dipping below previous baselines. Fable’s edge comes down to three things: sharper intuition for data quirks, strict adherence to a “golden workflow” that starts in a clean semantic layer then cross-checks raw SQL results, and transparent assumption-setting throughout its analysis. In one example, Opus zeroes in on a data oddity as its main finding, while Fable leads with the correct result (SMB), then notes the quirk and offers an alternate interpretation. In another case, Fable detects a cents-for-dollars error in a refund table by validating SQL outputs against the semantic model—something Opus misses entirely. Hex’s team also built a tougher “Frontier” benchmark to capture longer-horizon, open-ended problems. Fable at Max Effort scores 58% there, a clear jump over other setups. One task asks for the single insight you’d regret not including in a board presentation; passing requires framing the implicit business decision, exploring multiple hypotheses, and catching a cost-of-goods anomaly in the orders table. Those are the qualities that set Fable apart, showing real progress on complex data analysis.

Ask about this article