Click any tag below to further narrow down your results
Links
This page describes the Artificial Analysis Decision Index, a weighted benchmark that scores small "decision models" on fast classification tasks like routing support tickets and rating urgency. It combines six evaluations, from decision-making and tool calling to hallucination and answer invariance, into one composite score.
- Decision models answer every question in a single call with a probability for each option, which keeps them cheap and fast and makes calibration measurable.
- The index weights AA-Paperclip (document extraction as one decision) at 30%, with five other evaluations at 10–15% each.
- Accuracy counts the top-probability option as the answer, and the page also reports Brier scores, time per task, and cost per 1M tasks.
Google's new Decisions API lets developers get probability scores for closed-set choices like routing, policy checks, and grading, instead of generating text token by token. It's built on the Gemini Decision 1 model and aimed at fast, cheap control-plane decisions.
- Gemini Decision 1 scores candidate options in a single forward pass, returning probabilities and confidence scores at 150-250 ms end-to-end latency.
- Pricing is $0.10 per 1M input tokens with zero output tokens billed, and Google says it's nearly 5x cheaper than Gemini 3.5 Flash-Lite with the Interactions API on the shared benchmark suite.
- Each request can carry one multimodal input (text, images, audio, video, PDFs up to 256K tokens) and up to 1,000 questions, with yes/no, multiple-choice, and rubric-score types that return confidence values so ambiguous cases can go to a fallback or human.
The author compares newer decision models against LLM judges for automated evals on a leasing-agent failure task. The main takeaway is that the default 0.50 threshold often leaves performance on the table, and that the scores need tuning and calibration before you trust them.
- The default 0.50 decision threshold can miss nearly all failures; tuning it per model, using F1 with an 80% precision/recall floor, improves results.
- OpenAI Decisions had the best separation (F1 about 0.87) but cost about 4.5 times more than Perplexity Decider (F1 about 0.79) per 100 eval runs.
- Raw model probabilities are often poorly calibrated; Platt scaling helped Perplexity, Liquid d1, and Clef Flash, but made OpenAI's slightly worse.
Kev is a family of small language models (0.8B to 9B parameters) built on Qwen3.5 that make yes/no, multiple-choice, and rating decisions on text inputs. You can run them locally on your own hardware with an API compatible with TypeSafe's System One, plus there's a web playground for testing.
- Three model sizes available with published accuracy metrics: Kev-9B reaches 85.2% accuracy on unseen datasets and trails the proprietary Jev model by only 3.5 points, while Kev-4B and Kev-0.8B offer smaller footprints for different use cases.
- All models run on consumer hardware (CUDA, ROCm, Apple Silicon) with the 4B and 9B versions fitting on a 32GB Mac in bf16 precision, plus you can train your own weights using provided code and evaluation data.
- Probabilities are calibrated by default and the models output confidence scores alongside answers, reducing confident errors to 4% on new data compared to 8.7% without calibration.
Using LLMs directly as classifiers is frustrating because they can't be calibrated, don't reliably use all available data, and are hard to improve systematically. Instead, treat LLM outputs as features in a logistic regression or other ML model, which solves these problems while keeping the LLM's signal.
- Direct LLM classification fails on three critical fronts: you can't calibrate confidence scores or adjust precision-recall tradeoffs, the model may ignore structured data or context you provide, and you have no visibility into what the LLM actually used to make its decision.
- Wrapping an LLM verdict in logistic regression automatically fixes calibration (empirically matching true probabilities), lets you add other features, and gives interpretability about how much the LLM contributes to the final decision.
- On an irony detection task, logistic regression on the raw LLM verdict improved Brier score from 0.259 to 0.175 just by calibrating; adding more LLM-extracted features (token probabilities, multiple reasoning passes) provides a clear path to further improvement that prompt-tweaking doesn't offer.
AnalystGym offers a two-minute daily drill focused on analytical judgment rather than technical skills—the part of data work that AI can't handle. You rate your confidence, get feedback on your reasoning, and build a streak by showing up consistently.
- AI handles execution well (queries, transformations, calculations), but you still own the judgment calls: interpreting sudden metric changes, handling stakeholder requests, and defining metrics that hold up under pressure
- Four rotating drill formats (Lesson, Rapid Fire, Currency, The Rule) train different skills: careful reading, pattern recognition, time budgeting, and precise rule-writing
- Getting answers wrong is part of the learning—the value is in the reps and calibrating your confidence, not in being right every time
This article evaluates how new frontier models—Claude Fable 5, Opus 4.8, and GPT-5.5—perform on the long-horizon FrogsGame post-training task. Fable 5 stands out by programmatically generating high-quality supervision traces via a backtracking algorithm, combining them with RL, and reliably improving the base model where others still struggle with noisy data and evals.
- Claude Fable 5 won by generating correct solution traces via a backtracking algorithm rather than fine-tuning on the base model's noisy outputs, then combining that SFT data with RL within a 20-hour budget.
- Opus 4.8 tripled its predecessor's performance (30.9% pass@4) and GPT-5.5 reached 9.7%, but both still relied on the base model's flawed self-generated data rather than an independent correct-answer source.
- Fable 5's self-assessment was far better calibrated (1.2x overconfidence vs. Opus 4.8's 4.9x) and it used nearly the full time budget while producing 3x more diverse training data.
- Evaluation itself remains a weak point across all models—small sample sizes and noisy metrics make it hard to reliably judge real progress on the task.