The catalog lists each known bot with its id, name, operator, category, User-Agent tokens and verification methods. It is built from the operators' official lists: OpenAI (GPTBot, OAI-SearchBot, ChatGPT-User), Google (common crawlers, special crawlers, user-triggered fetchers), Bing, Perplexity (bot and user), Apple, Common Crawl, DuckDuckGo (DuckDuckBot and DuckAssistBot), Anthropic, the networks announced for Meta (AS32934), X (AS13414) and LinkedIn (AS14413), the documented reverse DNS suffixes, and the Web Bot Auth directories of ChatGPT and Browserbase.
- It ships with a built-in baseline, so verification works even before the first refresh.
- It is refreshed every 24 hours. For each source the last good copy and the last attempt are kept.
- A download that would leave a source with less than half of its entries is not used: the last good copy stays and the error is recorded.
Read it
The catalog feeds the Crawler pickers of AI Crawl Control → Settings → Crawler overrides and the filters of every report.
openwaf-cloud ctl ai catalogcurl -sS "https://dash.openwaf.dev/v1/orgs/$ORG/ai-agents/catalog" -H "Authorization: Bearer $OPENWAF_TOKEN"The answer has the catalog version, the bots and the state of each source; IP ranges and keys are not included.
ai_agents_catalog.
Use the bot ids of the catalog (for example gptbot, ccbot, googlebot) in per-bot overrides and in report filters.