Probably raised $9 million to build an AI system that catches hallucinations and factual errors before they reach users. Their data-science tool wraps LLM outputs in a deterministic validator “mech suit,” letting it run smaller models locally while ensuring each answer matches the source data.

• Probably raised $9M from a16z to build a validator system that blocks LLM outputs unless they match source data exactly, aiming for 99.99% accuracy.
• This validation approach lets them use models "four classes weaker" than frontier LLMs, cheap enough to run on a desktop instead of a GPU farm.
• Elias argues big AI labs won't build this themselves because their revenue model benefits from users paying per interaction, including ones spent correcting errors.

This article examines the reliability issues of large language models (LLMs) used in AI, highlighting their tendency to hallucinate and produce incorrect information. New research indicates that these problems stem from the models' inherent design, raising concerns about their suitability for high-stakes applications like law and accounting. Investors may need to reconsider the viability of AI business models given these risks.

• Hallucination rates rise sharply with input length: GLM 4.5 went from 1.2% errors at 32K words to 3.2% at 128K, and some models hallucinated in most cases at 200K words.
• Tsinghua research suggests hallucination-causing neurons are baked in during initial training, making the flaw structural rather than a fixable bug.
• NYT investigation found LLMs generating tax form errors serious enough to risk legal consequences like tax evasion, casting doubt on their use in high-stakes fields like law and accounting.
• Fixes for hallucination are being explored but likely years away, undercutting near-term AI business models built on high-reliability use cases.