With AI Crawl Control on, each request gets exactly one category.
| Category | Dashboard | What it is |
|---|---|---|
ai_training | AI Crawler | Crawlers that collect content to train models (GPTBot, ClaudeBot, CCBot, Google-Extended…). |
ai_search | AI Search | Crawlers of AI search engines that index pages to cite them (OAI-SearchBot, PerplexityBot…). |
ai_user_agent | AI Assistant | Agents that fetch a page because a person just asked an assistant (ChatGPT-User, Perplexity-User, Web Bot Auth signatures). |
search_engine | Search Engine Crawler | Classic search engines (Googlebot, Bingbot, Applebot, DuckDuckBot…). |
unverified_automation | Unverified automation | Automation with no verifiable identity: headless browsers, Playwright, python-requests, curl/, an empty User-Agent, scanner behaviour, clients that followed the trap. |
spoofed | Spoofed | Claims to be a catalog bot, and verification says otherwise. |
human | Human | Everything else. Always allowed; not configurable. |
Extra policy keys
Besides the six configurable categories, the policy has five keys for situations that can apply on top of the category. When several apply, the strongest action wins.
| Key | Dashboard | When it applies |
|---|---|---|
unverified_bot | Crawlers that can't be verified | The User-Agent names a catalog bot that publishes no IP ranges, reverse DNS or signatures (ClaudeBot, Bytespider…). |
trap_hit | Followed the trap | The client (IP or TLS fingerprint) followed the trap link before. |
secret_probe | Secret probing | The path looks for secrets: .env and .env.*, terraform.tfstate*, /@fs/, serviceaccount/token, .git/config, .git/HEAD, .aws/credentials, id_rsa, wp-config.php.*, .npmrc, docker-compose*.yml, .DS_Store… |
robots_violation | robots.txt violations | A catalog bot requests a path your robots.txt disallows for it (RFC 9309: the longest rule wins, Allow on a tie). |
agent_burst | Composed burst | A code-mode agent fires many distinct endpoints within seconds, walks through ids or paginates systematically. |
The edge reads the robots.txt responses that pass through it (and fetches it from your origin when it hasn't seen one for a name in 6 hours) to detect violations.
Actions
| Action | Dashboard | Effect in enforce mode |
|---|---|---|
allow | Allow | Count only. |
log | Log | Count and record a notable event. |
challenge | Challenge | The browser challenge for HTML GET navigations; 403 JSON otherwise. |
block | Block | The block response: 403 and the block page by default. |
tarpit | Tarpit | A slow maze of generated pages for HTML GET; 403 otherwise. Needs tarpit.enabled. |
GET /robots.txt is never challenged or blocked.