How To Use Decision Models for Evals – Hamel’s Blog
Short summary
The author compares newer decision models against LLM judges for automated evals on a leasing-agent failure task. The main takeaway is that the default 0.50 threshold often leaves performance on the table, and that the scores need tuning and calibration before you trust them.
- The default 0.50 decision threshold can miss nearly all failures; tuning it per model, using F1 with an 80% precision/recall floor, improves results.
- OpenAI Decisions had the best separation (F1 about 0.87) but cost about 4.5 times more than Perplexity Decider (F1 about 0.79) per 100 eval runs.
- Raw model probabilities are often poorly calibrated; Platt scaling helped Perplexity, Liquid d1, and Clef Flash, but made OpenAI's slightly worse.
Full summary
Decision models are small classifiers that return a 0-to-1 score estimating how likely a piece of text is to be a failure, and they're pitched as a cheap, fast alternative to LLM judges for automated evals. The author tested six of them (Jev 1.13, Perplexity Decider 1.1 27B, Liquid d1 and d1-ultra, Cloudflare Clef Flash, and OpenAI Decisions on GPT-6 Luna) on 100 labeled traces from an apartment leasing agent, where the job was to flag failed bookings or missed handoffs to a human. The main point is that the default 0.50 decision threshold is often wrong for a given task. A model that scores every real failure at 0.40 and every passing trace at 0.10 would classify perfectly at 0.30 and catch nothing at 0.50. The author recommends tuning the threshold against your own labeled data, and gives a concrete target: for an alerting system, at least 80% on both precision and recall.
On that test, OpenAI Decisions separated the classes best and had the highest F1 (0.87 after tuning), but at 3.72 cents per 100 eval runs it cost about 4.5 times as much as Perplexity Decider, which came in at 0.82 cents and an F1 of 0.79. Liquid d1 and Cloudflare Clef Flash fell below the 80% minimums even after tuning, while Perplexity, Liquid d1-ultra, and OpenAI cleared them. Tuning helped the strong models only modestly, but it rescued or sank the weaker ones, which is the more useful lesson. The author also points out that equal weighting via F1 is an assumption; F-beta lets you lean toward precision or recall depending on what a false alarm or a missed failure actually costs you. Latency was left out because it depends on conditions outside the experiment.
The last section asks whether the scores can be read as literal probabilities, so that a 0.80 means roughly an 80% chance of failure. That only holds if the model is well calibrated, and the author checks this with a reliability chart grouping about 20 traces per score bin, where points above the diagonal mean the model underestimates risk and points below mean it overestimates. The practical takeaway is to treat raw scores with caution until you've measured calibration on your own traces. I'd want to read the calibration results themselves before deciding how much to trust these numbers, since the excerpt cuts off just as that analysis begins.