Click any tag below to further narrow down your results
+ llm feature engineering
(1)
+ logistic regression
(1)
+ prompt engineering
(1)
+ decision models
(1)
+ decision thresholds
(1)
+ evals
(1)
+ llm judges
(1)
+ gemini decisions api
(1)
+ gemini decision 1
(1)
+ model routing
(1)
+ multimodal
(1)
+ decision model
(1)
+ ai evaluation
(1)
+ benchmark
(1)
+ llm
(1)
Links
This page describes the Artificial Analysis Decision Index, a weighted benchmark that scores small "decision models" on fast classification tasks like routing support tickets and rating urgency. It combines six evaluations, from decision-making and tool calling to hallucination and answer invariance, into one composite score.
- Decision models answer every question in a single call with a probability for each option, which keeps them cheap and fast and makes calibration measurable.
- The index weights AA-Paperclip (document extraction as one decision) at 30%, with five other evaluations at 10–15% each.
- Accuracy counts the top-probability option as the answer, and the page also reports Brier scores, time per task, and cost per 1M tasks.
Google's new Decisions API lets developers get probability scores for closed-set choices like routing, policy checks, and grading, instead of generating text token by token. It's built on the Gemini Decision 1 model and aimed at fast, cheap control-plane decisions.
- Gemini Decision 1 scores candidate options in a single forward pass, returning probabilities and confidence scores at 150-250 ms end-to-end latency.
- Pricing is $0.10 per 1M input tokens with zero output tokens billed, and Google says it's nearly 5x cheaper than Gemini 3.5 Flash-Lite with the Interactions API on the shared benchmark suite.
- Each request can carry one multimodal input (text, images, audio, video, PDFs up to 256K tokens) and up to 1,000 questions, with yes/no, multiple-choice, and rubric-score types that return confidence values so ambiguous cases can go to a fallback or human.
The author compares newer decision models against LLM judges for automated evals on a leasing-agent failure task. The main takeaway is that the default 0.50 threshold often leaves performance on the table, and that the scores need tuning and calibration before you trust them.
- The default 0.50 decision threshold can miss nearly all failures; tuning it per model, using F1 with an 80% precision/recall floor, improves results.
- OpenAI Decisions had the best separation (F1 about 0.87) but cost about 4.5 times more than Perplexity Decider (F1 about 0.79) per 100 eval runs.
- Raw model probabilities are often poorly calibrated; Platt scaling helped Perplexity, Liquid d1, and Clef Flash, but made OpenAI's slightly worse.
Using LLMs directly as classifiers is frustrating because they can't be calibrated, don't reliably use all available data, and are hard to improve systematically. Instead, treat LLM outputs as features in a logistic regression or other ML model, which solves these problems while keeping the LLM's signal.
- Direct LLM classification fails on three critical fronts: you can't calibrate confidence scores or adjust precision-recall tradeoffs, the model may ignore structured data or context you provide, and you have no visibility into what the LLM actually used to make its decision.
- Wrapping an LLM verdict in logistic regression automatically fixes calibration (empirically matching true probabilities), lets you add other features, and gives interpretability about how much the LLM contributes to the final decision.
- On an irony detection task, logistic regression on the raw LLM verdict improved Brier score from 0.259 to 0.175 just by calibrating; adding more LLM-extracted features (token probabilities, multiple reasoning passes) provides a clear path to further improvement that prompt-tweaking doesn't offer.