ai_agents is part of a zone's edge settings (GET/PATCH /v1/orgs/{org}/zones/{zone}/edge) and of each name's (hosts.<name>.ai_agents). Settings are validated even while AI Crawl Control is off.
| Field | Default | Values |
|---|---|---|
enabled | false | Classify requests and apply the policy. |
mode | log | log (count what would happen) or enforce. |
spoofed_challenge | log | enforce: only verified crawlers skip the browser challenge. log: keep the User-Agent exemption and record it. Independent of mode. |
policy.<key> | see defaults | allow, log, challenge, block or tarpit for ai_training, ai_search, ai_user_agent, search_engine, unverified_automation, spoofed, unverified_bot, trap_hit, secret_probe, robots_violation, agent_burst. |
template | — | Write-only: search_ok_block_training, verified_only or block_unverified_automation. Replaces the whole policy. |
bot_overrides | [] | Up to 100 {"bot_id", "action"}. |
trap.enabled | false | Insert the canary link. |
tarpit.enabled | false | Allow the tarpit action. |
robots.enabled | false | Prepend the managed section to robots.txt. |
robots.content_signals.search | yes | yes, no or "" (not stated). |
robots.content_signals.ai_input | yes | yes, no or "". |
robots.content_signals.ai_train | no | yes, no or "". |
markdown.enabled | false | Markdown for agents. |
block_response.status | 403 | 401, 402, 403, 404, 410, 429, 451 or 503. |
block_response.message | "" | Plain text up to 1024 bytes; "" = the site's block page. |
endpoints | [] | Up to 20 AI endpoints. Endpoints with kind: "mcp" belong to MCP Shield. |
Block response
A blocked request gets block_response.status. With a message, the body is that text (text/plain); with 429 or 503 the answer also carries Retry-After: 3600. Without a message, the site's block page is used with that status.
How a PATCH changes it
ai_agents in a PATCH …/edge is a JSON merge patch (RFC 7396):
- objects merge field by field, and the policy key by key;
- lists (
bot_overrides,endpoints) are replaced whole; - a field set to
nullgoes back to its default;"ai_agents": nullrestores every default; templatesets the template's whole policy first;policykeys in the same patch apply on top.
{"ai_agents": {"enabled": true, "mode": "enforce", "template": "search_ok_block_training"}}Per name
hosts.<name>.ai_agents replaces the zone's settings for that name as a whole. The first change to a name's ai_agents starts from a copy of the zone's, so the fields you leave out keep the zone's values. After that, later zone changes no longer reach the name. To follow the zone again, remove it: {"hosts": {"shop.example.com": {"ai_agents": null}}} (Use the domain's settings again in the dashboard).
CLI shortcuts
| Argument | Effect |
|---|---|
ai_agents.enabled=true | Any field by its dotted path. |
ai_agents.template=verified_only | Checked before calling the API. |
ai_agents.bot_overrides=gptbot:block,ccbot:allow | bot:action pairs; "" = none. |
ai_agents.endpoints:='[…]' | JSON value. |
--host shop.example.com | The name's own settings. |