# robots.txt and Content-Signal

> Let the edge prepend a managed section to your robots.txt with Disallow rules for the bots you block and a Content-Signal line with your preferences.

With `robots.enabled` (**Bot Preference Sync** in the dashboard), the edge prepends a managed section to the `robots.txt` your origin serves. It is added as the response is served: your origin and the cache are not changed.

```text
# OpenWAF AI agents (managed)
User-agent: gptbot
User-agent: ccbot
Disallow: /

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Disallow: /__waf/
# End OpenWAF AI agents
```

- The first group lists the bots whose effective action is `block` or `tarpit`, with `Disallow: /`. It is omitted when no bot is blocked.
- The `User-agent: *` group carries the `Content-Signal` line ([contentsignals.org](https://contentsignals.org)) and `Disallow: /__waf/`.
- If your origin answers `robots.txt` with 404, the edge serves the managed section alone.
- `GET /robots.txt` is never challenged or blocked by AI Crawl Control.

## Content signals

| Field | Default | Says whether your content may be used |
|---|---|---|
| `robots.content_signals.search` | `yes` | For search results. |
| `robots.content_signals.ai_input` | `yes` | As input to AI answers (RAG, grounding). |
| `robots.content_signals.ai_train` | `no` | To train models. |

Each value is `yes`, `no` or `""` (not stated: left out of the line). With all three empty, there is no `Content-Signal` line.

## Turn it on

**Dashboard**

**AI Crawl Control → Overview → Settings → Bot Preference Sync**: **Prepend the managed section to robots.txt**, choose the **Content signals** and check the **Preview**.
**CLI**

```sh
openwaf-cloud ctl zones set example.com ai_agents.robots.enabled=true ai_agents.robots.content_signals.ai_input=no \
  --reason "Publish AI preferences"
```
**API**

```json
{"ai_agents": {"robots": {"enabled": true, "content_signals": {"search": "yes", "ai_input": "no", "ai_train": "no"}}}}
```
**MCP**

`ai_agents_settings` with `"patch": {"robots": {"enabled": true}}`.


The **Signals** report shows, per name, how often `robots.txt` was fetched, whether it declares `Content-Signal`, and which bots fetched disallowed paths. See [Report views](https://developers.openwaf.dev/ai-crawl-control/report-views/).