AI Crawl Control classifies each request to a proxied name into a category, verifies crawlers that claim to be a known bot, and applies the action you choose for each category or each bot: allow, log, challenge, block or tarpit. It is off by default, can be set per zone or per name and has two modes: log (count what would happen) and enforce (apply it).
It also helps the AI agents you do want: it can add your preferences to robots.txt with a Content-Signal line, serve Markdown to agents that ask for it, and scan a name for agent readiness.
Get startedturn it on in log mode, read the report, then enforce.Categoriesthe seven categories and the extra policy keys.Bot verificationIP ranges, reverse DNS and Web Bot Auth, and what makes a crawler spoofed.Bot catalogthe verified bots, their operators and their official sources.Set the policytemplates, actions per category and per-bot overrides.robots.txt and Content-Signalprepend a managed section to your robots.txt.Trap and tarpitcatch automated readers and waste their time.Markdown for agentsanswer
Accept: text/markdown with Markdown.Protect your AI endpointsinjection detectors and limits for your own chat APIs.Agent readinessscore how ready a name is for AI agents.Settings referenceevery ai_agents field and its default.Report viewsoverview, crawlers, spoofers, optimization, signals, traps, bursts, endpoints and events.Where to find it
| Interface | Settings | Reports |
|---|---|---|
| Dashboard | Domain → AI Crawl Control → Overview → Settings | AI Crawl Control: Overview, Metrics, Security, Optimization, Signals, Traps, AI endpoints; Agent readiness |
| CLI | zones set … ai_agents.* | ai report, ai catalog, ai readiness |
| API | PATCH /v1/orgs/{org}/zones/{zone}/edge (ai_agents) | GET /v1/orgs/{org}/zones/{zone}/ai-agents/* |
| MCP | ai_agents_settings | ai_agents_report, ai_agents_catalog, agent_readiness_scan |
For your own MCP servers, see MCP Shield.