GitHub - mintlify/agent-traffic: Tell agents, AI crawlers, search crawlers and bots apart from people
Mintlify built agent-traffic, a dataset and small classifier that tells AI agents, crawlers, bots and people apart from web requests. It checks claimed identities against operators' published IP ranges to catch impersonators.
- Agent traffic to documentation sites is growing fast, with coding agents like Claude Code and Cursor fetching docs for users, alongside AI training crawlers.
- The classifier verifies crawlers like GPTBot against OpenAI's published IP ranges and flags requests that claim a crawler identity from the wrong address.
- Agents that disguise themselves with a browser user agent can still be caught by an Accept header that prefers markdown over HTML, which browsers never send.
- Some signals are blind spots: agentic browsers running inside a person's own Chrome look identical to a human, and logs that keep only the user agent miss the header check.
Mintlify runs documentation for thousands of companies, and its sites now get a growing share of requests from AI agents rather than people reading in a browser. Claude Code pulls pages into coding sessions, ChatGPT answers questions from docs, Cursor checks API references before writing calls. The problem is that the bot lists they started with just matched user agent substrings, lumped all automation into one bucket, and trusted whatever a request said it was. So a request claiming to be GPTBot might really be OpenAI, or might be someone borrowing the name, and there was no way to tell which kind of traffic a given hit represented.
Their answer is agent-traffic, an open-source package and dataset. It holds a curated list of user agents, headers, and official verification sources for real agents and crawlers, plus a zero-dependency classifier that runs in Node, Bun, Deno, Cloudflare Workers, and Vercel Edge. The classifier sorts requests into categories: agent (an AI product fetching for a person), ai-crawler (training or AI search collection), crawler (search engines), bot (everything else automated), human, and unknown. Verification is the part that matters most. Where an operator publishes IP ranges, as OpenAI, Anthropic, Google, Perplexity, Apple, Microsoft, Common Crawl, and DuckDuckGo do, the classifier checks the client IP against them and marks spoofers as impersonators. Each result returns a verified flag (true, false, or null when it can't be checked) and a list of reasons, so you can see why it reached its verdict.
Two details are worth knowing before you use it. First, agents that borrow a browser user agent get caught through the Accept header: anything ranking text/markdown above text/html is almost certainly not a browser, since browsers never send that. Second, if your logs only keep the user agent, you'll still identify known agents, but disguised ones will look like people unless you start logging Accept too. The package also ships a Claude Code and Cursor skill for setting up classification in your own stack, and a Cloudflare Worker example that serves markdown to agents and HTML to everyone else. The dataset is importable as plain JSON, so it works outside JavaScript too.