More on the topic...
Generating detailed summary...
Failed to generate summary. Please try again.
Armin Ronacher ran into a Pi edit tool glitch while testing Anthropic’s Claude models. With both Opus 4.8 and Sonnet 5, the models would generate perfectly correct “oldText” and “newText” values but tack on extra, made-up fields—things like “requireUnique,” “type,” “oldText2” or even “event.0.additionalProperties.” Pi’s validator rejects those malformed calls, so the model has to try again. Strangely, older Anthropic releases never showed this problem; only the newest state-of-the-art versions do.
The root seems to lie in how tool calls get signaled. Anthropic wraps calls in an XML-like “antml” format, then parses out JSON for nested arrays. You can either generate JSON freely and validate it after the fact or use grammar-aware sampling to block invalid tokens. Claude’s latest models aren’t constrained that way, so they drift from the schema under complex, multi-turn histories. In single-turn prompts they behave fine; in an agentic session with file reads and edits they’ll invent random keys up to 20% of the time.
Ronacher’s theory: post-training on Claude Code’s own tools hardened the models around that client’s forgiving harness. Claude Code silently retries, coerces types, strips unknown keys and fixes Unicode—all without penalizing the model. So during reinforcement learning, slightly malformed calls still succeed and get rewarded. When you pair a new model with a different schema, it fights you harder because its prior expects the Claude Code shape.
That shift shows how a stronger model can actually be less flexible with external tools. Opus 4.5 once adapted well to arbitrary edit schemas; Opus 4.8 now struggles unless you mask out bad tokens or strip out thinking blocks from history. If you rely on an undocumented, closed-source harness at Anthropic, your own tool formats risk becoming off-distribution—and you won’t know until your calls start failing.
Questions about this article
No questions yet.