@ryanbai - Devin Offers $10M Productivity
@ryanbai on X Cognition is betting its Devin coding agent delivers about four times the engineering output customers pay for, and backing that with a guarantee of up to $10M per customer. The claim rests on an automated estimator agent that scored 258 real enterprise sessions, with a reported correlation of 0.74 against human judgment.
- The estimator was built on 258 human-annotated sessions from 126 users, filtered to keep only work that led to merged PRs or clearly valuable outcomes.
- Lines of code changed correlate with output, but much of the signal sits outside the diff, in investigation and diagnosis, which Cognition says sometimes produces no code at all.
- Cognition says historical data shows roughly 4x return versus cost, and it will fund usage if Devin ever delivers less value than customers pay for.
Cognition, the company behind the coding agent Devin, is betting up to $10 million per customer that Devin produces more engineering output than customers pay for. To back that claim, they built an estimator and tested it on a dataset of 258 sessions from 126 users across enterprise customers, which they describe as the first human-annotated dataset of enterprise coding agent traces. They filtered it down to productive work by dropping sessions where none of the pull requests were merged, and for sessions with no PRs at all, a classifier screened out the ones that didn't produce anything valuable.
The estimator is an agent that gets the user's messages, the PR (when one exists), the full agent trace, and codebase context from DeepWiki. Cognition tuned its prompt on 25 development sessions and added four principles to make it conservative. On a held-out set it reaches a 0.74 correlation on a log scale and looks unbiased. Individual estimates are noisy, but the company argues the errors cancel out in aggregate. The more interesting finding is what drives the signal. Simpler predictors, like total lines changed across PRs or the estimator seeing only the edit tool calls, do worse, which suggests much of the value sits outside the diff. Investigation, diagnosis, and reasoning carry real weight, and the most useful work sometimes yields no code at all.
Across historical customer data, Cognition says Devin delivers roughly four times what customers pay, valued at a conservative blended global wage. The AI Productivity Guarantee means that if Devin ever falls short, Cognition keeps funding usage until it catches up, capped at $10 million per customer. The full methodology and limitations are in their technical report at cognition.ai/blog/ai-productivity. Worth reading that before taking the 4x figure at face value, since the estimator's validation and the choice of wage benchmark are doing a lot of work in that number.