With robots.enabled (Bot Preference Sync in the dashboard), the edge prepends a managed section to the robots.txt your origin serves. It is added as the response is served: your origin and the cache are not changed.
robots.txt (managed section)
# OpenWAF AI agents (managed)
User-agent: gptbot
User-agent: ccbot
Disallow: /
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Disallow: /__waf/
# End OpenWAF AI agents- The first group lists the bots whose effective action is
blockortarpit, withDisallow: /. It is omitted when no bot is blocked. - The
User-agent: *group carries theContent-Signalline (contentsignals.org (external link)) andDisallow: /__waf/. - If your origin answers
robots.txtwith 404, the edge serves the managed section alone. GET /robots.txtis never challenged or blocked by AI Crawl Control.
Content signals
| Field | Default | Says whether your content may be used |
|---|---|---|
robots.content_signals.search | yes | For search results. |
robots.content_signals.ai_input | yes | As input to AI answers (RAG, grounding). |
robots.content_signals.ai_train | no | To train models. |
Each value is yes, no or "" (not stated: left out of the line). With all three empty, there is no Content-Signal line.
Turn it on
AI Crawl Control → Overview → Settings → Bot Preference Sync: Prepend the managed section to robots.txt, choose the Content signals and check the Preview.
openwaf-cloud ctl zones set example.com ai_agents.robots.enabled=true ai_agents.robots.content_signals.ai_input=no \
--reason "Publish AI preferences"{"ai_agents": {"robots": {"enabled": true, "content_signals": {"search": "yes", "ai_input": "no", "ai_train": "no"}}}}ai_agents_settings with "patch": {"robots": {"enabled": true}}.
The Signals report shows, per name, how often robots.txt was fetched, whether it declares Content-Signal, and which bots fetched disallowed paths. See Report views.