A User-Agent is just text. A request that claims to be a bot of the catalog counts as verified only if one of that bot's verification methods succeeds.
| Method | Dashboard | Verified when |
|---|---|---|
ip | IP ranges | The client IP is in the ranges the operator publishes. |
rdns | Reverse DNS | The PTR name of the IP ends in one of the bot's suffixes (such as .googlebot.com) and that name resolves back to the same IP (forward-confirmed reverse DNS). |
web_bot_auth | Web Bot Auth | The request carries valid Signature-Agent, Signature-Input (tag web-bot-auth, Ed25519) and Signature headers (HTTP Message Signatures, RFC 9421), checked against the keys the agent publishes in /.well-known/http-message-signatures-directory. |
Spoofed and unverifiable
- A User-Agent that claims a catalog bot and fails verification (IP outside its ranges, reverse DNS missing or not matching) is spoofed.
- A DNS error doesn't make a request spoofed: it stays unverified, with the signal
rdns_error. - An invalid Web Bot Auth signature leaves the signal
wba_invalid. - Bots that publish no ranges, reverse DNS or signatures (ClaudeBot, Bytespider…) are unverifiable: they keep their category with
verified=false, theunverified_botpolicy key also applies (the strongest action wins) and they are never exempt from anything.
Spoofed crawlers and the challenge
Without verification, saying "Googlebot" used to be enough to skip the browser challenge. With spoofed_challenge: enforce, only verified crawlers skip it: fakes get the same challenge as anyone. With log (the default), the edge keeps the User-Agent exemption but records that it would have challenged. spoofed_challenge is independent of mode.
In the dashboard: Settings → Protection → Block fake crawlers at the challenge.
Verified crawlers never get the trap link, and are never challenged or tarpitted for following it.