Click any tag below to further narrow down your results
Links
The author compares newer decision models against LLM judges for automated evals on a leasing-agent failure task. The main takeaway is that the default 0.50 threshold often leaves performance on the table, and that the scores need tuning and calibration before you trust them.
- The default 0.50 decision threshold can miss nearly all failures; tuning it per model, using F1 with an 80% precision/recall floor, improves results.
- OpenAI Decisions had the best separation (F1 about 0.87) but cost about 4.5 times more than Perplexity Decider (F1 about 0.79) per 100 eval runs.
- Raw model probabilities are often poorly calibrated; Platt scaling helped Perplexity, Liquid d1, and Clef Flash, but made OpenAI's slightly worse.
Using LLMs directly as classifiers is frustrating because they can't be calibrated, don't reliably use all available data, and are hard to improve systematically. Instead, treat LLM outputs as features in a logistic regression or other ML model, which solves these problems while keeping the LLM's signal.
- Direct LLM classification fails on three critical fronts: you can't calibrate confidence scores or adjust precision-recall tradeoffs, the model may ignore structured data or context you provide, and you have no visibility into what the LLM actually used to make its decision.
- Wrapping an LLM verdict in logistic regression automatically fixes calibration (empirically matching true probabilities), lets you add other features, and gives interpretability about how much the LLM contributes to the final decision.
- On an irony detection task, logistic regression on the raw LLM verdict improved Brier score from 0.259 to 0.175 just by calibrating; adding more LLM-extracted features (token probabilities, multiple reasoning passes) provides a clear path to further improvement that prompt-tweaking doesn't offer.