Most AI agent benchmarks measure average success rate—how often an agent succeeds across multiple runs—but hide a serious reliability problem. A ReAct agent using GPT-4.1 succeeds 77.4% of the time on average, but only succeeds on all five consecutive runs for 53% of tasks. That 24-point gap means nearly a quarter of the benchmark consists of tasks the agent can solve sometimes and fail other times, with nothing about the task actually changing. This isn't a capability problem that a bigger model fixes. It's about decision uncertainty: when an LLM picks which action to take next, that choice emerges from a probability distribution over tokens. Sharp distributions put almost all their weight on one token, so the same choice wins run after run. Flat distributions spread comparable probability across several near-tied options, making them vulnerable to tiny perturbations—floating-point rounding, request batching, platform quirks—that can flip which token wins. Since agent workflows chain dozens of decisions together, small per-step flipping compounds into large consistency gaps. IBM Research built the Consistency Analyzer to find and fix these flip-prone steps. The tool takes a single recorded trajectory and resamples each decision point with controlled variation, measuring how much the model's output actually changes at that step. This happens offline with one extra model call per decision (using k=5 completions by default), replayed against the already-recorded context—no re-running the full task. The analyzer flags high-uncertainty decisions and converts them into consistency guidelines, which are just targeted instructions injected back at inference time. A real example: after the agent struggled inconsistently on counting checkbox markers in notes, it generated "When counting checkbox-style markers in note content, use a line-anchored regex match rather than a plain substring count." These aren't task-specific tricks; they target instability patterns that surface across many tasks. Testing on AppWorld with 168 tasks showed the consistency gap shrinking from 24.4 points down to 12 points. Pass^5 (all five runs succeeding) jumped from 53.0% to 69.0%, while average accuracy actually improved slightly from 77.4% to 81.0%. Medium and hard tasks saw the biggest gains: +22.9 points and +14.3 points respectively. The system halved the reliability problem without sacrificing average performance, turning an agent that looks capable into one you can actually count on.