Artificial Analysis Decision Index v2.1 | Artificial Analysis

Open original

This page describes the Artificial Analysis Decision Index, a weighted benchmark that scores small "decision models" on fast classification tasks like routing support tickets and rating urgency. It combines six evaluations, from decision-making and tool calling to hallucination and answer invariance, into one composite score.

  • Decision models answer every question in a single call with a probability for each option, which keeps them cheap and fast and makes calibration measurable.
  • The index weights AA-Paperclip (document extraction as one decision) at 30%, with five other evaluations at 10–15% each.
  • Accuracy counts the top-probability option as the answer, and the page also reports Brier scores, time per task, and cost per 1M tasks.

The Artificial Analysis Decision Index is a composite score built from six evaluations, each weighted differently. The heaviest slice goes to AA-Paperclip at 30%, which tests decision-making and extraction by scoring a model's 15 answers on a document as one decision. The other five are EvalSafe-O*NET-AA (15%, decision-making and judgement), When2Call (10%, tool calling), AA-Omniscience-Decision v1.1 (15%, knowledge and hallucination), HLE-MultiChoice v1.1 (15%, reasoning), and AA-Symmetry v1.1 (15%, decision invariance, scored by a Symmetry Score rather than a Decision Score). What makes this index different from a typical leaderboard is that it measures classification-style decisions, not free-form generation. The model gets a fixed set of options, returns a probability for each, and answers every question in a single call. That makes it cheap and fast to run, and it lets the team check calibration, not just whether the top pick was right. The worked example in the article shows how this looks in practice. A customer writes that payouts have been failing for three days and rent is due tomorrow. The model has to choose a team to route to (Billing, Technical, or Account), say whether the message is urgent, and rate how upset the customer is on a scale from Calm to Angry. Every answer comes back as a probability distribution across options. Accuracy is scored by whether the highest-probability option is the correct one, while calibration is measured with Brier scores, so a model that is confidently wrong gets punished more than one that hedges sensibly. The page also reports time per task and cost per million tasks, and plots the index against cost, which is the part a product team would likely care about most when picking a model for triage at volume. The article is mostly a dashboard and methodology page, so the real value is in the framing: this is a benchmark for cheap, high-volume routing and classification work, not for open-ended reasoning or coding. The page also links out to a long list of other Artificial Analysis evaluations covering agentic work, coding, medical and legal tasks, and multimodal reasoning. Those sit outside the Decision Index itself, so if you are deciding whether to trust the composite number, the weights and the six component evals are the part worth reading closely.

Ask about this article