AI robots.txt Checker
See what your robots.txt tells 14 AI crawlers, from OpenAI and Perplexity to Google and Common Crawl, and which rules are costing you AI visibility.
Not all AI bots are the same
Search bots
OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot and Bingbot build the indexes AI answers come from. Block them and you can't be cited.
Live fetchers
ChatGPT-User, Perplexity-User and Claude-User open a page when someone asks about it. Blocking them breaks answers about your own pages.
Training bots
GPTBot, ClaudeBot, Google-Extended, CCBot and others collect data for future models. Blocking them is a policy choice that doesn't affect search.
Questions
Frequently asked
Should I block GPTBot?+
Blocking GPTBot stops OpenAI from training on your pages. It doesn't remove you from ChatGPT search, which uses OAI-SearchBot and ChatGPT-User. Block training bots if that's your policy, but keep the search and live-fetch bots allowed if you want to be cited.
Which bots decide whether AI assistants cite me?+
The search and live-fetch bots: OAI-SearchBot and ChatGPT-User (OpenAI), PerplexityBot and Perplexity-User, Claude-SearchBot and Claude-User (Anthropic), Googlebot (Google Search, AI Overviews and AI Mode) and Bingbot (Bing and Copilot).
Does Google-Extended affect AI Overviews?+
No. Google-Extended controls whether Gemini models can use your content; AI Overviews and AI Mode come from Google Search, which uses Googlebot.
robots.txt allows the bots but they still can't get in. Why?+
Bot protection in your CDN or firewall (Cloudflare, Akamai, a WAF) can refuse AI crawlers even when robots.txt allows them. Our AI readiness checker tests that too.
Fixing your site is step one. Is AI recommending you?
Track how ChatGPT, Perplexity, Gemini and Google's AI answer the questions your buyers ask, against your competitors, every week.