MLCR-AA: Medical Long Context Reasoning Benchmark | Artificial Analysis

Open original

This is a leaderboard page for MLCR-AA, Artificial Analysis's version of Wisedocs' open benchmark, which tests whether models can reason across long, fragmented medical records the way insurance claims reviewers do. The top model, Claude Fable 5.2, scores a 43.9% pass rate on the hardest question tiers.

  • Each case is a synthetic medical record of roughly 25,000 to 64,000 tokens, and questions are drawn from the two hardest of six difficulty tiers.
  • A panel of three LLM judges grades each answer against expert ground truth and the source record; a response passes only if a majority find it complete and free of contradictions.
  • Empty responses count as failures, and the score is standalone, not part of the Intelligence Index.

MLCR-AA is Artificial Analysis' version of MLCR, an open benchmark from Wisedocs that checks whether models can do the kind of work claims professionals do on medical files: pulling together chronology, causation, treatment patterns, and claim relevance across long, fragmented records. Each case is a synthetic medical record of roughly 25,000 to 64,000 tokens. The benchmark has six difficulty tiers, but Artificial Analysis only runs the two hardest, expert synthesis and compound multi-part reasoning, and reports results over those sets. It is a standalone evaluation and doesn't feed into the Intelligence Index. Scoring works like this. Three LLM judges compare each response against expert-annotated ground truth and the source record. A response passes only if a majority of judges each find it complete and free of any claim that contradicts the record. Empty responses aren't judged and just count as failures. The headline number is pass rate, with accuracy and completeness reported alongside it. At the time of the page, Claude Fable 5.2 (Max, Default Fallback) led with a 43.9% pass rate, which tells you how hard this is: even the top model fails more than half of these cases. The page also breaks out cost per task, output tokens per task, time per task, and score against release date, so you can see whether the best scores are coming at a sensible price. That matters for the stated audience, organizations running off-the-shelf models over medical records, insurance claims, and other long-document healthcare work. A model that clears 44% of expert-level cases is still going to need human review on anything consequential, and the cost and speed charts are the place to judge whether a given model is worth running at that volume.

Ask about this article