5 min read
|
Saved February 14, 2026
|
Copied!
Do you care about this?
This article discusses a proposed registry for bots and agents that enables website operators to discover and verify public keys for cryptographic authentication. It outlines the implementation of Web Bot Auth and the signature-agent card format, aimed at fostering a trustworthy ecosystem for web traffic management.
If you do, here's more
As bots and agents begin to cryptographically sign their requests, website operators face challenges in verifying their identities. Discovering the public keys of these agents, especially when dealing with thousands or millions of them, is a significant issue. Cloudflare proposes a registry format to help website operators easily find and authenticate these agents. This registry would function similarly to existing IP address lists and robots.txt files, allowing operators to access a curated list of URLs pointing to public keys for various bots and agents.
The article highlights a protocol called Web Bot Auth, which enables bot developers to sign their requests. Several companies, including Vercel and Shopify, have implemented this protocol. However, the challenge remains for smaller operators to discover the public keys of known crawlers. Proposed solutions include a new HTTP header, Signature-Agent, to point to endpoints hosting these keys. The article emphasizes the need for a structured registry that maintains control over allowed traffic while promoting an open ecosystem.
Cloudflare also introduces the signature-agent card format, which adds metadata to the existing key directory, providing essential information about each bot or agent, such as contact methods and expected crawl rates. This format enhances the utility of the registry, making it easier for operators to understand the context and intentions of the agents interacting with their sites. The article concludes with practical guidance on how to implement and curate these registries, encouraging collaboration and transparency among operators.
Questions about this article
No questions yet.