Heidi Health CEO: Frontier Models Have Plateaued

Open original

Tom Kelly, CEO of Heidi Health, claims that frontier AI models stopped improving on clinical note-taking tasks around Claude Sonnet 3.5, forcing the company to build its own fine-tuned models instead. This directly contradicts a June Nature Medicine study showing frontier models beating specialized clinical AI, but both claims are actually true—they're measuring different things (raw intelligence vs. production reliability).

  • Kelly argues frontier models now "get different, not better" on Heidi's specific tasks—what matters is consistency and rule-following, not raw intelligence. Heidi built synthetic training data from 3 million weekly visits to create cheaper, more reliable alternatives to API-based models.
  • An unpublished Heidi randomized trial found that only 1 in 10 doctors catch dangerous hallucinations in AI-generated clinical notes, with real incidents like hallucinated chemotherapy drug names nearly identical to real ones.
  • Kelly predicts 90% of the 126+ AI scribes in the market will fail, with survivors being those that prove consistent safety performance. He also said Heidi will "for sure" become the first FDA-regulated AI scribe in the US.

Tom Kelly, CEO of Heidi Health (an AI scribe company processing 3 million clinical visits weekly), claims frontier models stopped improving on his core tasks around Claude Sonnet 3.5. After that point, newer releases just get "different, not better"—smarter models actually perform worse because they break the rigid rules and templates doctors have tuned. So Heidi went back to building its own fine-tuned models using synthetic data generated by frontier models as supervisors, then refined against real clinician preferences. Kelly claims this approach is both cheaper (speech-to-text on their hardware costs 100 times less than APIs) and more reliable for the narrow, repetitive task of turning patient conversations into medical notes without hallucinating drug names. This directly contradicts what the author argued two months earlier: a June Nature Medicine study showed frontier models beating specialized clinical AI tools on medical knowledge benchmarks. But they're answering different questions. The Nature study measured raw medical knowledge and answer quality—territory where Claude and ChatGPT dominate. Kelly's plateau is about production reliability on one specific task done thousands of times daily with zero tolerance for invented medications and consistent cost. On that axis, raw intelligence stopped mattering around Sonnet 3.5; what matters now is consistency, rule-following, and cost. Both stories can be true: the frontier keeps getting smarter, but it stopped getting better for Heidi's particular problem. Kelly's defensibility rests on three legs: preference data from 3 million weekly visits that enterprise agreements prohibit frontier labs from training on, regulatory approval (Heidi is a licensed medical device in the UK), and economics (hyperscalers won't undercut their own cloud margins). The author's pushback: all three are rented. One board meeting changes focus, and if Big AI decides clinical AI is worth the effort, Heidi's moat collapses. Kelly acknowledged this honestly, saying he'd switch to a better frontier model if one emerged. He also cited an unpublished randomized trial showing only 1 in 10 doctors catch dangerous hallucinations in AI-generated notes—a safety problem the industry barely discusses.

Ask about this article