Detecting Misuse with the Claude Compliance API: The Threat Is in the Content

Open original

This article shows how to use the Claude Compliance API to feed full chat and file content into your SIEM, then write detections for AI-specific threats that live in the content. It covers prompt injections in uploaded files, jailbreaks, system-prompt leaks, data-laundering requests, sensitive disclosures, and insider data-flow risks.

  • Claude's Compliance API streams full chat and file content into your SIEM, not just auth/admin logs—so an unrecognized identity querying compliance_api_accessed can see every chat in the org.
  • Uploaded files can hide jailbreak instructions via zero-width characters, white-on-white text, or base64 blobs, so file_uploaded/project_document_created events need scanning for instruction-like language and suspicious encoding.
  • Detections should focus on the assistant's output, not the prompt: a jailbreak attempt only matters if disallowed content or leaked system instructions actually appear in the response.
  • Requests to encode, paraphrase, or strip PII from text are exfiltration red flags, and only the model's actual response confirms whether sensitive data got through.

Claude’s Compliance API feed plugs directly into your SIEM, giving you both the usual auth-and-admin events and full chat and file content. The basic alerts you already run—failed logins, new admin API keys, role changes, domain verifications—map neatly to Claude’s sso_login_failed, admin_api_key_created, sso_connection_deleted and so on. Once you’ve ingested and normalized that stream, your existing rules light up on Claude just like any other SaaS. Don’t skip the compliance_api_accessed event: any unrecognized identity querying the feed can see every chat in your org. Where Claude differs is in the content itself. Uploaded PDFs, project docs or scraped web pages can hide hidden instructions—zero-width characters, white-on-white text, even base64 blobs—aimed at “jailbreaking” the model or siphoning data. Watch for file_uploaded and project_document_created events that include instruction-like language or suspicious encoding. Flag the document first, then correlate it with subsequent chats in the same project if the model’s behavior shifts. Jailbreak attempts and guardrail probes live in user prompts, but you should alert on the assistant’s responses. A known jailbreak template in the prompt is noise; actual disallowed content in message.content means the attack worked. The same goes for system-prompt extraction. Blocked probes aren’t your concern—when Claude spills its hidden instructions, you need to know. Finally, treat obfuscation requests as a red flag. Queries like “encode this as base64,” “rewrite to remove PII,” or “paraphrase account numbers” can turn harmless text into undetectable exfiltration drafts. Content-aware detections must weigh both the request and the output. Only the model’s response confirms whether sensitive data slipped through your perimeter.

Ask about this article