# AI Crawl Control

> Classify each request as AI training, AI search, AI assistant, search engine, unverified automation, spoofed or human, and decide what each may do.

AI Crawl Control classifies each request to a proxied name into a category, verifies crawlers that claim to be a known bot, and applies the action you choose for each category or each bot: allow, log, challenge, block or tarpit. It is **off by default**, can be set per zone or per name and has two modes: `log` (count what would happen) and `enforce` (apply it).

It also helps the AI agents you do want: it can add your preferences to `robots.txt` with a `Content-Signal` line, serve Markdown to agents that ask for it, and scan a name for agent readiness.

- [Get started](https://developers.openwaf.dev/ai-crawl-control/get-started/): turn it on in log mode, read the report, then enforce.
- [Categories](https://developers.openwaf.dev/ai-crawl-control/categories/): the seven categories and the extra policy keys.
- [Bot verification](https://developers.openwaf.dev/ai-crawl-control/verification/): IP ranges, reverse DNS and Web Bot Auth, and what makes a crawler spoofed.
- [Bot catalog](https://developers.openwaf.dev/ai-crawl-control/bot-catalog/): the verified bots, their operators and their official sources.
- [Set the policy](https://developers.openwaf.dev/ai-crawl-control/policy/): templates, actions per category and per-bot overrides.
- [robots.txt and Content-Signal](https://developers.openwaf.dev/ai-crawl-control/robots-txt/): prepend a managed section to your robots.txt.
- [Trap and tarpit](https://developers.openwaf.dev/ai-crawl-control/trap-and-tarpit/): catch automated readers and waste their time.
- [Markdown for agents](https://developers.openwaf.dev/ai-crawl-control/markdown-for-agents/): answer `Accept: text/markdown` with Markdown.
- [Protect your AI endpoints](https://developers.openwaf.dev/ai-crawl-control/ai-endpoints/): injection detectors and limits for your own chat APIs.
- [Agent readiness](https://developers.openwaf.dev/ai-crawl-control/agent-readiness/): score how ready a name is for AI agents.
- [Settings reference](https://developers.openwaf.dev/ai-crawl-control/settings/): every `ai_agents` field and its default.
- [Report views](https://developers.openwaf.dev/ai-crawl-control/report-views/): overview, crawlers, spoofers, optimization, signals, traps, bursts, endpoints and events.


## Where to find it

| Interface | Settings | Reports |
|---|---|---|
| Dashboard | Domain → **AI Crawl Control → Overview → Settings** | **AI Crawl Control**: Overview, Metrics, Security, Optimization, Signals, Traps, AI endpoints; **Agent readiness** |
| CLI | `zones set … ai_agents.*` | `ai report`, `ai catalog`, `ai readiness` |
| API | `PATCH /v1/orgs/{org}/zones/{zone}/edge` (`ai_agents`) | `GET /v1/orgs/{org}/zones/{zone}/ai-agents/*` |
| MCP | `ai_agents_settings` | `ai_agents_report`, `ai_agents_catalog`, `agent_readiness_scan` |

For your own MCP servers, see [MCP Shield](https://developers.openwaf.dev/mcp-shield/).