AI agents that succeed 77% of the time on average only succeed consistently on 53% of tasks — a reliability gap hidden by standard benchmarks. IBM Research built a diagnostic tool that identifies unstable decision points in an agent's reasoning and converts them into guidelines that cut this gap in half without sacrificing overall accuracy.

• Standard metrics (Mean@k) mask a consistency problem: tasks an agent can solve sometimes but not others, with nothing about the task changing. On hard tasks, the gap between average success rate and consistent success rate reaches 30 percentage points.
• The Consistency Analyzer pinpoints flip-prone steps by resampling each decision point in a recorded trace with a single model call (requesting 5 completions), identifying where the model's output distribution is too flat to be reliable.
• Consistency guidelines derived from these unstable steps cut the gap roughly in half (24.4pp → 12.0pp) and generalize to related tasks, lifting Pass^5 by +16pp on the same task and +13pp on similar tasks without reducing average accuracy.