Skip to content
SEO Rank Fixer
Sign inStart free

AI Crawler Checker: Is GPTBot or ClaudeBot Blocked?

Check if robots.txt or Cloudflare blocks GPTBot, ClaudeBot, PerplexityBot and 12 more AI crawlers from your site, with copy-paste fixes. Free.

Updated

Checks robots.txt and sends a live request as each crawler. Takes up to 20 seconds.

What this AI crawler checker tests

An AI assistant can only quote and cite pages its crawler is allowed to fetch. Two separate things decide that, and site owners often only know about one of them, so the checker tests both for 15 crawlers:

  • robots.txt. For each crawler it finds the group of rules that applies (the group naming that crawler, otherwise User-agent: *) and the rule that decides your page, using Google's matching: the longest matching rule wins and Allow wins a tie. You see the exact line.
  • The live answer. It requests the page once as an ordinary browser and once with each crawler's user-agent string, and compares the answers. A 403, 429 or 503, or a challenge page such as Cloudflare's "Just a moment..." (marked by the cf-mitigated: challenge header), shown to a crawler but not to the browser means a firewall, CDN or security plugin is turning that crawler away, whatever robots.txt says.

How to read the results

  • Allowed: robots.txt lets the crawler in and the server answered it like a browser.
  • Blocked by robots.txt: a Disallow rule covers this page for that crawler. Well-behaved crawlers will not fetch it.
  • Likely blocked: robots.txt allows it, but the server refused or challenged the crawler's user-agent while serving the browser.
  • Inconclusive: the request failed, or the site blocked our browser test too.

One honest limit: a user-agent test cannot fully reproduce the real crawler. Cloudflare and similar services verify real crawlers by their IP addresses or by signed requests, and our request comes from our own server. A firewall may let the real GPTBot in and challenge our imitation, or the other way round. Treat the live result as a strong hint and confirm it in your firewall's logs.

How to fix a blocked AI crawler

In robots.txt

If a crawler has its own group with Disallow: /, delete that line (or the group). If it is caught by your User-agent: * rules, give it its own group. Remember that a crawler with its own group ignores the * group entirely, so copy in any Disallow lines you still want:

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
Disallow: /checkout/

To keep training crawlers out while staying visible in AI search, block only the training tokens:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

Test any change before you upload it with the robots.txt tester.

In Cloudflare

  • Open Security Settings, filter by Bot traffic and choose Configure AI bot policies. Policies are set per kind of crawler (Search, Agent, Training): Allow, Block, or Block on pages with ads. Cloudflare's AI bot documentation notes that new domains created from 15 September 2026 block Training and Agent crawlers on pages with ads by default, so check this even if you never changed it.
  • In the same place, Enable Bot Preference Sync lets Cloudflare write AI rules into your robots.txt (its managed robots.txt). If the checker says your robots.txt is partly managed by Cloudflare, change those rules there.
  • Bot fight mode challenges automated traffic, and WAF skip rules do not exempt anything from it. If it is on and AI crawlers are blocked, it is a likely cause.
  • AI Crawl Control (formerly AI Audit) shows which AI crawlers visit, lets you allow or block each one, and flags crawlers that ignore your robots.txt.

On other hosts

Look for bot or user-agent rules in your host's firewall, in other CDNs (Akamai, Fastly, Sucuri) and in WordPress security plugins. Many have a one-click "block AI bots" option that is easy to forget.

Related free tools

Common questions

How do I know if Cloudflare is blocking AI crawlers on my site?

Run the checker above: if a crawler gets a 403, a 429 or a Cloudflare challenge page while a normal browser gets the page, a firewall rule is turning it away. Then look in Cloudflare under Security Settings, filtered by Bot traffic: the AI bot policies, Bot fight mode and your custom WAF rules are the usual causes. Cloudflare's security events log shows which rule acted on which request.

Should I block GPTBot and ClaudeBot?

It depends on what you want. GPTBot and ClaudeBot collect pages to train models; blocking them keeps your content out of future training data. OAI-SearchBot, Claude-SearchBot and PerplexityBot find pages to cite in AI answers; blocking those removes you from answers that could send visitors. Many sites block training crawlers and allow search crawlers, which robots.txt lets you do crawler by crawler.

Does blocking Google-Extended remove my site from Google Search or AI Overviews?

No. Google-Extended is a robots.txt token only, with no crawler of its own. It controls whether Google may use pages Googlebot already crawled for Gemini model training and grounding. Google says it does not affect inclusion or ranking in Search. AI Overviews are part of Search and use Googlebot.

Do AI crawlers obey robots.txt?

The major ones say they do for their automatic crawlers: OpenAI, Anthropic, Perplexity, Google and Apple all document robots.txt tokens. Fetches triggered by a person in a chat are different: OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally does not follow robots.txt. Some crawlers, such as Bytespider, have been reported ignoring it.

Why does the checker say 'inconclusive'?

Either the request failed or timed out, or the site also turned away our normal-browser test. In the second case the site blocks requests from data-centre servers in general, so a difference between crawlers cannot be measured from outside. The robots.txt verdicts still hold; check your firewall logs for the rest.