AI crawler checker
The short answer
An AI crawler checker shows whether the robots that read web pages for ChatGPT, Claude, Perplexity and Copilot can open your site. Type your website. We read its robots.txt, the file that tells robots what they may open, then request the home page under the name of 9 crawlers, because the server or the firewall can still refuse a crawler that robots.txt allows.
Last updated:
| Crawler | robots.txt | Server | Result |
|---|---|---|---|
| GPTBotOpenAICollects pages to train OpenAI’s models | |||
| OAI-SearchBotOpenAIReads pages so ChatGPT search can show them | |||
| ChatGPT-UserOpenAIOpens a page when a ChatGPT user asks for it | |||
| ClaudeBotAnthropicCollects pages to train Anthropic’s models | |||
| Claude-SearchBotAnthropicReads pages so Claude’s search can show them | |||
| Claude-UserAnthropicOpens a page when a Claude user asks for it | |||
| PerplexityBotPerplexityReads pages so Perplexity can show them in its answers | |||
| Perplexity-UserPerplexityOpens a page when a Perplexity user asks for it | |||
| BingbotMicrosoftReads pages for Bing, the search Microsoft Copilot uses | |||
| Google-ExtendedGoogleA robots.txt rule with no robot of its own. It lets Google use your pages for Gemini |
Questions about AI crawlers
What is an AI crawler?
An AI crawler is a robot that an AI company sends to read web pages. There are 3 kinds. Some collect pages to train a model (GPTBot, ClaudeBot). Some read pages so the engine can show them in its answers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Bingbot). Some open a single page when a user asks for it (ChatGPT-User, Claude-User, Perplexity-User).
Which AI crawlers should I allow?
To be named in AI answers, allow the crawlers that read pages for live answers: OAI-SearchBot and ChatGPT-User for ChatGPT, Claude-SearchBot and Claude-User for Claude, PerplexityBot and Perplexity-User for Perplexity, Bingbot for Copilot. GPTBot and ClaudeBot collect pages to train the models. Blocking them keeps you in live answers, and the next model learns nothing about you from your own site.
My robots.txt allows a crawler. Why is it still refused?
A robots.txt file only states what robots may read. The firewall in front of your site decides who gets the page, and some refuse AI crawlers by default. This checker requests your home page under each crawler’s name: 200 means served, 403 means refused. The AI readiness checker shows what a crawler can read once it is in.
How do I unblock an AI crawler?
If robots.txt blocks it, delete the Disallow line under the crawler’s name in the file, or under “User-agent: *” when the line applies to every robot. If the server refuses it, open the bot settings of your firewall or hosting provider and allow that crawler. On Cloudflare, the setting is called “Block AI bots”.
Can a result be wrong?
Yes. The main case: we send each request from our own server with the crawler’s name on it. Some firewalls also check where a request comes from: they refuse our test and still serve the real crawler. If a refusal surprises you, look at your firewall’s bot settings before you change anything.
Does the checker test my whole site?
The checker tests the home page. A firewall rule or a “Disallow: /” line applies to every page, so both show up here. A rule that blocks 1 folder only does not.