Interpreting Pangram
Armin Ronacher tests Pangram, an AI detection tool, by having an LLM generate text based on a detailed prompt, then manually rewriting it from scratch without AI assistance. Both versions get flagged as 100% AI-generated, suggesting that detection tools struggle with text that originated from LLM-structured prompts even after substantial human revision.
- Pangram claims extremely low false positive rates (0.0041% false AI accusations), yet flags clearly human-authored text as entirely AI when the author used an LLM to develop the initial structure
- The rewritten text shares only 50% similarity to the LLM output with no identical sentences, yet still scores as 100% AI, indicating detection models may penalize writing that follows LLM-suggested structural patterns
- Authors who rely on LLMs for organizing ideas face a credibility problem: their work gets flagged as AI regardless of how much manual editing and rewriting happens afterward
Armin Ronacher ran an experiment with Pangram, an AI detector that claims a 0.0041% false positive rate for flagging human text as AI. After David Sacks tweeted something that Pangram immediately labeled as AI-generated, Sacks dismissed the detector as bogus. Ronacher decided to test whether Pangram's accuracy holds up in practice by using Claude Opus 5 to regenerate Sacks's tweet based on a detailed prompt about "pacing the frontier"—a reference to Dario Amodei's post on slowing AI development and Sam Altman's agreement. The AI output read entirely synthetic, exactly as Pangram predicted.
The real test came when Ronacher manually rewrote the AI-generated text from scratch without using any LLM. He kept the same structure and core arguments but rewrote every sentence, achieving only 50% similarity to the original AI version. He then ran spell-check with an LLM but didn't regenerate any content. Pangram still flagged this entirely human-written text as 100% AI. This matters because Ronacher suspects the problem isn't Pangram's accuracy so much as how LLM-assisted writing works: when you use an AI tool to structure your thoughts, the underlying patterns remain recognizable to detectors, even if every word is yours.
The experiment exposes a real gap between Pangram's claimed performance and what happens in messy reality. A detector with a 0.34% miss rate and 0.0041% false positive rate sounds bulletproof on paper, but those numbers assume clean scenarios. Once human writers lean on LLMs for scaffolding—prompts, outlines, spell-check—the detector can't distinguish between "AI-assisted human writing" and "fully synthetic text." The article cuts off mid-thought, but the implication is clear: these detectors may be technically competent at their narrow job while remaining useless for what people actually want to know.