FAQ

Common questions about AI Crawl Control: defaults, verification, search engines, robots.txt and what happens to people.

Is it on by default?

No. AI Crawl Control is in beta, off by default, and starts in log mode when you turn it on: nothing is blocked until you switch to enforce.

Will blocking AI crawlers hurt my search ranking?

The default policy and every template allow search_engine and ai_search. Search engines are verified, so blocking spoofed doesn't affect the real Googlebot or Bingbot.

How do you know a bot is who it says it is?

Each catalog bot is checked against its operator's published IP ranges, forward-confirmed reverse DNS or Web Bot Auth signatures. A User-Agent that claims to be Googlebot from a cloud server is spoofed. See Bot verification.

What about bots that don't publish how to verify them?

They keep their category with verified=false and the unverified_bot key also applies; the strongest action wins.

Can people be blocked?

The human category is always allowed. People are only affected by the other keys if they behave like automation: for example, requesting .env files (secret_probe).

Does robots.txt stop crawlers?

Only the ones that respect it. The managed section states your preferences; the policy enforces them at the edge, and robots_violation catches catalog bots that ignore your robots.txt.

How long is the data kept?

30 days.

Last updated:

↑↓ to navigate↵ to openesc to close