Measuring alignment eval tells
METR ran investigator agents through capability and alignment evals to see whether models can spot which kind they're in. Current models can, mostly by noticing the environment looks fake or that the task is a honeypot built to elicit bad behavior.
- Investigator agents have gotten better at separating alignment evals from capability evals over the past year, and Opus 5.5 ranked every alignment eval above every capability eval on their dataset.
- The tells come in two kinds: environment-level tells like obviously mocked services, and task-level tells like unethical shortcuts or high-stakes tradeoffs deliberately built in to incriminate the model.
- If models notice they're being tested, alignment evals may overstate how well a model behaves, though the authors only measured awareness, not whether it changes behavior.
Alignment evals have a problem: models may be able to tell they're being tested. The METR note starts from a result by Needham and Edkins (2025) showing models sometimes recognize an alignment eval as one. The usual way to build an alignment eval is to mock up a capabilities eval or a real deployment, say a sandbox-escape scenario modeled on the Hugging Face incident. The trouble is that a toy version can tip the model off. If the agent knows it's in a test, it may behave better than it would in deployment, which means the eval overstates alignment. The authors split these giveaways into two kinds: environment-level tells, like obviously mocked services or implausible infrastructure (fixable by making the simulation more realistic), and task-level tells, like unethical shortcuts or high-stakes conflicts that are built in on purpose to catch bad behavior and so can't simply be engineered away.
To measure how detectable these tells are, the team ran "investigator agents" inside 19 alignment evals and 36 capability evals (22 Time Horizons tasks and 14 Terminal-Bench 4.0 tasks). The alignment set includes inverse rubric optimization honeypots, game-playing honeypots where cheating beats a strong chess engine, physical-safety scenarios, and Petri auditing seeds. Each investigator got the same tools as a normal agent and was asked two things: its direct percent-chance estimate of being in an alignment eval, and an open-ended question about the task's purpose, graded 0 to 100 by an LLM judge. Each task-judge pair ran three times. The authors flag a real limitation here: asking the model whether it's being tested primes it to think about testing, so the measurement comes from comparing alignment and capability tasks under the same questioning rather than from any claim about what the model would conclude unprompted.
The headline result is that discrimination has improved over the past year. On this dataset Opus 5.5 ranks every alignment eval above every capability eval on both its direct and open-ended scores, which gives an AUC of 1.0. The note is careful about what that does and doesn't mean. It doesn't prove alignment evals are impossible to disguise, only that the standard construction method yields evals current models can often spot. The investigators mostly picked up on environments that "look fake" and on tasks that read as honeypots. Those are the tells the team wants to understand before deciding whether any observed alignment is real. Whether these tells actually change behavior during an eval is a separate question the post explicitly leaves open.