# Categories

> The seven request categories and the extra policy keys for unverifiable bots, trap followers, secret probes, robots.txt violations and bursts.

With AI Crawl Control on, each request gets exactly one category.

| Category | Dashboard | What it is |
|---|---|---|
| `ai_training` | AI Crawler | Crawlers that collect content to train models (GPTBot, ClaudeBot, CCBot, Google-Extended…). |
| `ai_search` | AI Search | Crawlers of AI search engines that index pages to cite them (OAI-SearchBot, PerplexityBot…). |
| `ai_user_agent` | AI Assistant | Agents that fetch a page because a person just asked an assistant (ChatGPT-User, Perplexity-User, Web Bot Auth signatures). |
| `search_engine` | Search Engine Crawler | Classic search engines (Googlebot, Bingbot, Applebot, DuckDuckBot…). |
| `unverified_automation` | Unverified automation | Automation with no verifiable identity: headless browsers, Playwright, `python-requests`, `curl/`, an empty User-Agent, scanner behaviour, clients that followed the trap. |
| `spoofed` | Spoofed | Claims to be a catalog bot, and [verification](https://developers.openwaf.dev/ai-crawl-control/verification/) says otherwise. |
| `human` | Human | Everything else. Always allowed; not configurable. |

## Extra policy keys

Besides the six configurable categories, the policy has five keys for situations that can apply on top of the category. When several apply, the strongest action wins.

| Key | Dashboard | When it applies |
|---|---|---|
| `unverified_bot` | Crawlers that can't be verified | The User-Agent names a catalog bot that publishes no IP ranges, reverse DNS or signatures (ClaudeBot, Bytespider…). |
| `trap_hit` | Followed the trap | The client (IP or TLS fingerprint) followed the [trap link](https://developers.openwaf.dev/ai-crawl-control/trap-and-tarpit/) before. |
| `secret_probe` | Secret probing | The path looks for secrets: `.env` and `.env.*`, `terraform.tfstate*`, `/@fs/`, `serviceaccount/token`, `.git/config`, `.git/HEAD`, `.aws/credentials`, `id_rsa`, `wp-config.php.*`, `.npmrc`, `docker-compose*.yml`, `.DS_Store`… |
| `robots_violation` | robots.txt violations | A catalog bot requests a path your `robots.txt` disallows for it (RFC 9309: the longest rule wins, `Allow` on a tie). |
| `agent_burst` | Composed burst | A code-mode agent fires many distinct endpoints within seconds, walks through ids or paginates systematically. |

The edge reads the `robots.txt` responses that pass through it (and fetches it from your origin when it hasn't seen one for a name in 6 hours) to detect violations.

## Actions

| Action | Dashboard | Effect in enforce mode |
|---|---|---|
| `allow` | Allow | Count only. |
| `log` | Log | Count and record a notable event. |
| `challenge` | Challenge | The [browser challenge](https://developers.openwaf.dev/waf/browser-challenge/) for HTML `GET` navigations; `403` JSON otherwise. |
| `block` | Block | The [block response](https://developers.openwaf.dev/ai-crawl-control/settings/#block-response): 403 and the block page by default. |
| `tarpit` | Tarpit | A slow maze of generated pages for HTML `GET`; `403` otherwise. Needs `tarpit.enabled`. |

`GET /robots.txt` is never challenged or blocked.