Click any tag below to further narrow down your results
Links
AI agents that succeed 77% of the time on average only succeed consistently on 53% of tasks — a reliability gap hidden by standard benchmarks. IBM Research built a diagnostic tool that identifies unstable decision points in an agent's reasoning and converts them into guidelines that cut this gap in half without sacrificing overall accuracy.
- Standard metrics (Mean@k) mask a consistency problem: tasks an agent can solve sometimes but not others, with nothing about the task changing. On hard tasks, the gap between average success rate and consistent success rate reaches 30 percentage points.
- The Consistency Analyzer pinpoints flip-prone steps by resampling each decision point in a recorded trace with a single model call (requesting 5 completions), identifying where the model's output distribution is too flat to be reliable.
- Consistency guidelines derived from these unstable steps cut the gap roughly in half (24.4pp → 12.0pp) and generalize to related tasks, lifting Pass^5 by +16pp on the same task and +13pp on similar tasks without reducing average accuracy.
Hex built a suite of analytical evals to test data-analysis models and found Claude Fable 5 outperforms its Opus 4.x predecessors by 10–15%, nailing both semantically modeled and raw-data tasks with fewer mistakes. They’ve also designed a tougher “Frontier” benchmark for long-horizon, open-ended scenarios, where Fable 5’s careful assumptions and cross-checks boost its pass rate to around 58%.
- Claude Fable 5 beats Opus 4.7 by 10-15 points on Hex's core benchmarks, scoring 93%+ on Analytical Hard/Semantically Modeled tests and 65% on Semantically Unmodeled tasks, versus prior Opus versions' single-digit gains
- Fable's advantage comes from following a "golden workflow" (starting in the semantic layer, cross-checking raw SQL) and transparently stating assumptions, which lets it catch errors like a cents-for-dollars mistake that Opus misses
- On Hex's new "Frontier" benchmark for long-horizon, open-ended tasks, Fable at Max Effort hits 58% pass rate, notably outperforming other setups