Your Agent Aced the Task. Will It Do It Again?

Open original

AI agents that succeed 77% of the time on average only succeed consistently on 53% of tasks — a reliability gap hidden by standard benchmarks. IBM Research built a diagnostic tool that identifies unstable decision points in an agent's reasoning and converts them into guidelines that cut this gap in half without sacrificing overall accuracy.

  • Standard metrics (Mean@k) mask a consistency problem: tasks an agent can solve sometimes but not others, with nothing about the task changing. On hard tasks, the gap between average success rate and consistent success rate reaches 30 percentage points.
  • The Consistency Analyzer pinpoints flip-prone steps by resampling each decision point in a recorded trace with a single model call (requesting 5 completions), identifying where the model's output distribution is too flat to be reliable.
  • Consistency guidelines derived from these unstable steps cut the gap roughly in half (24.4pp → 12.0pp) and generalize to related tasks, lifting Pass^5 by +16pp on the same task and +13pp on similar tasks without reducing average accuracy.

Most AI agent benchmarks measure average success rate—how often an agent succeeds across multiple runs—but hide a serious reliability problem. A ReAct agent using GPT-4.1 succeeds 77.4% of the time on average, but only succeeds on all five consecutive runs for 53% of tasks. That 24-point gap means nearly a quarter of the benchmark consists of tasks the agent can solve sometimes and fail other times, with nothing about the task actually changing. This isn't a capability problem that a bigger model fixes. It's about decision uncertainty: when an LLM picks which action to take next, that choice emerges from a probability distribution over tokens. Sharp distributions put almost all their weight on one token, so the same choice wins run after run. Flat distributions spread comparable probability across several near-tied options, making them vulnerable to tiny perturbations—floating-point rounding, request batching, platform quirks—that can flip which token wins. Since agent workflows chain dozens of decisions together, small per-step flipping compounds into large consistency gaps. IBM Research built the Consistency Analyzer to find and fix these flip-prone steps. The tool takes a single recorded trajectory and resamples each decision point with controlled variation, measuring how much the model's output actually changes at that step. This happens offline with one extra model call per decision (using k=5 completions by default), replayed against the already-recorded context—no re-running the full task. The analyzer flags high-uncertainty decisions and converts them into consistency guidelines, which are just targeted instructions injected back at inference time. A real example: after the agent struggled inconsistently on counting checkbox markers in notes, it generated "When counting checkbox-style markers in note content, use a line-anchored regex match rather than a plain substring count." These aren't task-specific tricks; they target instability patterns that surface across many tasks. Testing on AppWorld with 168 tasks showed the consistency gap shrinking from 24.4 points down to 12 points. Pass^5 (all five runs succeeding) jumped from 53.0% to 69.0%, while average accuracy actually improved slightly from 77.4% to 81.0%. Medium and hard tasks saw the biggest gains: +22.9 points and +14.3 points respectively. The system halved the reliability problem without sacrificing average performance, turning an agent that looks capable into one you can actually count on.

Ask about this article