Why AI crawler access matters
AI assistants like ChatGPT, Claude and Perplexity answer many questions by searching the web and citing pages. If your robots.txt blocks their search crawlers, your pages can't be cited, no matter how good they are. Many sites block them by accident, often with a blanket rule copied from a template or a security plugin.
Training crawlers vs search crawlers
Most AI companies now run separate crawlers for different jobs, and you can treat them differently:
| Company | Training | Search / answers | User requests |
|---|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User |
| Perplexity | (none) | PerplexityBot | Perplexity-User |
| Google-Extended | Googlebot | (Googlebot) |
Blocking a training crawler keeps your content out of future model training. Blocking a search crawler removes you from that assistant's answers. If you want AI visibility, allow the search and user crawlers at the very least.
How the checker decides
It reads your live robots.txt and applies the same matching rules crawlers use: the most specific user-agent group applies, and inside it the longest matching rule wins, with Allow winning a tie. It then reports whether your homepage is allowed for each crawler and which rule decided it.
What robots.txt can't tell you
Some sites block AI bots at the firewall or CDN instead, for example with Cloudflare's AI bot blocking. Those blocks don't appear in robots.txt. If your robots.txt allows a crawler but you never appear in its answers, check your CDN and security settings too. My guide to ChatGPT SEO explains the other checks.