Prompt Injection Explained Through Role Tags
The article shows how LLMs use discrete role tags (like <user>, <think>, <tool>) to structure context but also rely on writing style, causing them to misclassify malicious commands in tool-tagged text as user instructions. Using linear probes to measure internal “roleness” (CoTness, Userness, etc.), the authors reveal why prompt injections succeed and suggest focusing research on the science of roles to defend against such attacks.
- LLMs classify roles more by writing style/content than by the actual tags surrounding text, so tool-tagged content that reads like a command gets internally treated as a user instruction
- Researchers used linear probes on hidden activations (wrapping neutral text in different role tags) to show this "content overrides tag" effect directly in the model's internal representations
- This explains why prompt injections achieve near-100% success against frontier models in practice even though those same models score well on static benchmarks that only test memorized attack patterns
- The fix isn't more attack examples but strengthening models' internal tracking of role boundaries—a "science of roles" approach to defense
LLMs treat every turn—system prompts, user requests, tool outputs, even their own hidden reasoning—as one long stream of tokens. Providers wrap those streams in tags like <system>, <user>, <assistant>, <think> and <tool> to impose structure. Each tag signals how the model should handle the following text: <user> commands guide actions, <think> marks private reasoning, <tool> flags external data you shouldn’t obey. Without those tags, the model just sees “token soup,” so moving text from one tag to another is the only clear-cut way to steer behavior.
Attackers exploit that reliance on tags with prompt injections. When an agent fetches a web page, the page content arrives inside <tool> tags, which should be treated as inert data. But human-crafted injections often read like legitimate user instructions, and models frequently misclassify tool‐tagged text as <user> prompts. Static benchmarks test against known attacks—models memorize patterns like “send your .env file” and block them—but real‐world adversaries can rephrase their payloads until the LLM misreads the role tags. That gap gives attackers near-100% success against frontier models, despite those same models acing benchmark suites.
To pinpoint that failure, researchers built “role probes.” They take neutral snippets—“Beginners BBQ Class!”—wrap each copy in a different role tag, then train a simple classifier on the model’s hidden activations to see which tag the model thinks each token belongs to. The results show that content often overrides tags: the model’s internal representation of a command buried in <tool> looks much more like a <user> command than like external data. That weak role perception explains why LLMs ignore tag boundaries, and it suggests that improving robustness against prompt injection will require strengthening the model’s ability to track and respect those discrete role signals.