Anthropic says it fixed Claude's writing. I ran the evals to check.

Open original

A researcher at Arize built a rigorous evaluation system to test whether Claude Opus 5.5 actually fixed the distinctive writing quirks that plague Claude outputs. The answer: it improved by 50%, but "fixed" overstates it — the model still relies heavily on rhetorical moves that make AI-generated text feel artificial.

  • Em dashes dropped dramatically (from 12.9 to 0.035 per 1,000 words), but other Claudisms only halved across six categories: salience flags, verdict intensifiers, signposts, contrast reframes, stock metaphors, and gotcha framing.
  • The real problem isn't specific phrases—it's rhetorical moves. Claude doesn't say "load-bearing" that often, but it constantly tells readers something matters instead of showing why, which no phrase list can catch.
  • Evaluating writing quality requires human annotation first, then building automated checks against those annotations. The researcher manually flagged 153 instances across 53,000 words to create ground truth before training an LLM judge.

Anthropic claimed that Claude Opus 5.5 fixed the model's notorious writing tics—the jargon-heavy, self-conscious phrasing that makes AI-generated text instantly recognizable on social media. To test this, the author built a proper evaluation framework: 20 research briefs across different genres and domains, ran both Opus 5 and Opus 5.5 twice each through identical prompts, then manually annotated the output for "Claudisms." The em dash problem is genuinely solved—Opus 5 used 12.9 per 1,000 words while Opus 5.5 had just two across 57,000 words total. Everything else improved by roughly 50%, dropping from 4.79 Claudisms per 1,000 words to 2.38. But here's the catch: the famous Claudisms everyone jokes about—"load-bearing," "delve," "crucially"—barely showed up. Only 9 of 153 flagged phrases matched the hand-written list. What actually dominated the output were rhetorical moves, not fixed phrases. Nearly half the flags were moments where the text *told* you something mattered instead of *showing* why. Others were contrast reframes like "It's a topology, not a genre," verdict intensifiers like "the honest answer," and teasing signposts like "Here's the part that surprises people." These aren't wording problems you can phrase-list away. The evaluator was built to catch these moves across six categories, using OpenAI's GPT-6-Luna as a span judge (so Claude doesn't grade itself) and normalizing for the fact that Opus 5.5 writes about 8% longer. The improvement is real and measurable, but the claim that Claude's writing got "fixed" is overstated. Fifty percent fewer Claudisms is better. It's not fixed.

Ask about this article