Is Cloudflare blocking ChatGPT and Claude from your site?

Cloudflare can block ChatGPT, Claude and other AI crawlers at the edge, before a request ever reaches your server, and on many zones that is now the default. Whether it blocks them on your site depends on three things: when the domain joined Cloudflare, the AI bot policies in your Security Settings, and whether Bot Fight Mode is challenging bots. You can check all three in about ten minutes and set it up so AI search engines can still read and cite you while training crawlers stay out.

Why would Cloudflare block AI crawlers on my site?

Cloudflare added a one-click option to block AI crawlers in September 2024. On July 1, 2025, it went further: in its press release the company said that “upon sign-up with Cloudflare, every new domain will now be asked if they want to allow AI crawlers,” with blocking as the starting point. The stated reason was that AI companies take far more content than they send back in clicks. Cloudflare called the day “Content Independence Day.”

A year later, Cloudflare replaced the single toggle. Its July 1, 2026 changelog lets every customer, including the Free plan, manage AI traffic by behavior, and the Block AI bots documentation marks the old “Block AI bots” option as deprecated from September 15, 2026. So if you set things up in 2025, the screen you remember may not be the one you see now.

None of this is a mistake on Cloudflare's part. It is a policy choice, and it is often the right one for publishers who live on ad revenue. The problem is that many small business owners never made the choice consciously, and then wonder why ChatGPT says it cannot open their site.

What do Cloudflare's Search, Agent and Training policies block?

Cloudflare classifies AI bots by what they do, not by who owns them. Each behavior has its own setting:

Each behavior can be set to Block (on all pages), Block on pages with ads (Cloudflare detects ad units and blocks only there), or Allow. The setting covers verified bots with that behavior plus unverified bots that Cloudflare classifies the same way.

Two details matter for AI visibility. First, from September 15, 2026, Cloudflare says mixed-purpose crawlers that do both search and training will be blocked by any setting meant to block training. Second, new domains onboarding from that date start with Training and Agent blocked on pages with ads, while Search stays allowed. Existing zones were given until that date to opt out of the new defaults, so look at what your zone actually has rather than assuming.

Which AI crawlers should I allow if I want to be cited?

OpenAI and Anthropic both split their crawlers by purpose, which is what makes a middle path possible. OpenAI's crawler documentation says OAI-SearchBot “is used to surface websites in search results in ChatGPT's search features,” GPTBot is for training, and ChatGPT-User handles actions a user starts. Anthropic's support article describes the same split: ClaudeBot for training, Claude-SearchBot to improve search results, and Claude-User for pages a person asks Claude to read, adding that disabling the last two “may reduce” your visibility in Claude's answers.

A sensible default for a business that wants to be recommended but not used for training:

If you also want to keep content out of Gemini training without touching Google Search, that is a robots.txt matter, not a Cloudflare one: Google-Extended is a robots.txt token that Googlebot obeys, not a separate crawler, so there is nothing for Cloudflare to block at the edge. The same goes for Applebot-Extended.

How do I check whether Cloudflare is blocking AI crawlers?

1. Open the AI bot policies. In the Cloudflare dashboard, pick your domain, go to Security, then Settings, filter by Bot traffic and choose Configure AI bot policies. Note what Search, Agent and Training are set to. On zones that still show the legacy option, look for Block AI bots on the same page.
2. Open AI Crawl Control. This section, formerly called AI Audit and available on every plan according to Cloudflare's docs, lists which AI services requested your pages, how often, and whether each was allowed or blocked. You can also set allow or block per crawler here.
3. Check Bot Fight Mode. Under Security, then Bots, see whether Bot Fight Mode (Free) or Super Bot Fight Mode (Pro and above) is on. Then look in Security Events or Security Analytics for challenges served to AI user agents.
4. Read the robots.txt crawlers actually get. Open https://yourdomain.com/robots.txt in a browser. If Cloudflare's managed robots.txt (Bot Preference Sync) is on, you will see its lines above your own.
5. Run an outside check. Our free AI crawler checker reads your robots.txt and reports which AI crawlers it lets in, crawler by crawler. It cannot see Cloudflare's edge decisions, which is why steps 1 to 3 come first.

Why doesn't testing with curl and a GPTBot user agent work?

It is tempting to run curl -A "GPTBot" https://yourdomain.com and read the status code. The result tells you very little. Cloudflare decides whether a request is the real GPTBot by checking where it comes from. Its verified bots page explains that a bot must identify itself reliably, for example through Web Bot Auth signatures, a published IP list or reverse DNS. OpenAI publishes its ranges (for GPTBot, openai.com/gptbot.json), and Anthropic publishes claude.com/crawling/bots.json.

Your laptop is not on those lists. A request from it that claims to be GPTBot is, from Cloudflare's point of view, an unverified bot pretending to be one. It may be allowed when the real crawler is blocked, or blocked when the real one gets through. The dashboard data from real crawler visits is the only trustworthy answer.

What does Cloudflare's managed robots.txt add?

When Bot Preference Sync is on and at least one AI policy blocks, Cloudflare prepends managed directives to your robots.txt. According to the managed robots.txt docs, the block starts with a Content Signals policy and then disallows known training crawlers. A typical result looks like this:

User-Agent: *
Content-signal: search=yes, ai-train=no, use=reference
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

# ...your own robots.txt follows here

Content-signal is Cloudflare's proposal for stating preferences in three categories: search, ai-input (using content in real-time AI answers) and ai-train, and Cloudflare is testing an optional use field on top. It is a stated preference, not an enforcement mechanism; the same docs say robots.txt compliance is voluntary. Cloudflare also warns that Google Search Console may report “Syntax not understood” for these lines, and says it has seen no effect on crawling.

Is Bot Fight Mode blocking AI crawlers?

Bot Fight Mode is the free plan's blunt instrument. Cloudflare's Bot Fight Mode docs say plainly: “You cannot bypass or skip Bot Fight Mode using WAF custom rules or Page Rules.” That means you cannot carve out an exception for a crawler you want. If your Security Events show AI crawlers being challenged, your options are to turn Bot Fight Mode off or move to a paid plan, where Super Bot Fight Mode runs on the Ruleset Engine and supports Skip rules, and lets you set Verified bots to Allow.

How do I let AI search crawlers in but keep training crawlers out?

1. In Configure AI bot policies, set Search to Allow and Agent to Allow.
2. Set Training to Block, or to Disallow if you only want a robots.txt signal. Decide whether you want this on all pages or only on pages with ads.
3. Turn on Bot Preference Sync if you want the matching rules published in robots.txt too.
4. In AI Crawl Control, confirm that OAI-SearchBot, Claude-SearchBot and the user-triggered agents show as allowed.
5. Make your own robots.txt agree with the edge settings, so the two never contradict each other. A version that matches the policy above:

   User-agent: OAI-SearchBot
   Allow: /
   
   User-agent: Claude-SearchBot
   Allow: /
   
   User-agent: PerplexityBot
   Allow: /
   
   User-agent: GPTBot
   Disallow: /
   
   User-agent: ClaudeBot
   Disallow: /
   
   User-agent: Google-Extended
   Disallow: /
   
   User-agent: *
   Allow: /
   
   Sitemap: https://yourdomain.com/sitemap.xml

6. Give it time. OpenAI says it can take about 24 hours from a robots.txt update for its search systems to adjust. Then ask ChatGPT to open one of your pages and see if it can.

You do not have to repeat this by hand. SEO Rank Fixer's AI visibility panel (free on every plan) checks whether GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot and other AI crawlers may read your site, and on Pro it also asks ChatGPT and Claude your customers' questions to see whether they name or cite you.

What about pay per crawl?

Cloudflare also offers pay per crawl, which lets a site charge AI crawlers for access instead of only allowing or blocking them. The AI Crawl Control documentation lists it as a private beta. It is mainly relevant to publishers with content AI companies want at scale. For a local business or a SaaS site that wants to be recommended, the question is the opposite one: how to make it as easy as possible for AI search crawlers to read you.

What to do next

Once crawlers can get in, the work moves to being worth citing. Our guide to getting recommended by ChatGPT covers what the answers draw on, and the GEO guide covers the wider picture. If you are weighing whether an llms.txt file helps, read what the evidence says first. For definitions of the terms used here, see the glossary.