ClaudeBot: Anthropic’s training crawler explained
ClaudeBot collects public pages for training Anthropic’s Claude models. How to block it or slow it with Crawl-delay, and how to verify it.
Updated
ClaudeBot is Anthropic’s crawler for collecting public web content that may be used to train Claude models. Anthropic says that when a site restricts ClaudeBot, it signals that the site’s future materials should be excluded from its AI model training datasets. Blocking it does not stop Claude’s search or user-requested fetches, which use separate agents.
| Owner | Anthropic |
|---|---|
| Purpose | Training |
| robots.txt token | ClaudeBot |
| Follows robots.txt | Yes, including Crawl-delay |
| Published IP ranges | claude.com/crawling/bots.json |
| If you block it | Your future pages are excluded from Anthropic’s training data. |
Anthropic’s help article does not publish ClaudeBot’s full user-agent string. Match on the token ClaudeBot.
What does ClaudeBot do?
Anthropic says ClaudeBot helps enhance the utility and safety of its generative AI models by collecting web content that could contribute to their training. Anthropic also states that its bots honor “do not crawl” signals in robots.txt and do not try to bypass CAPTCHAs or other anti-circumvention measures.
Anthropic runs two other agents: Claude-SearchBot for search quality and Claude-User for pages fetched when a user asks Claude a question.
How do I block ClaudeBot?
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /Anthropic asks you to add the rule to the robots.txt of every subdomain you want to opt out, since each subdomain has its own file.
How do I slow ClaudeBot down instead?
User-agent: ClaudeBot
Crawl-delay: 1Crawl-delay is not part of the robots.txt standard and Google ignores it, but Anthropic says it supports it for its bots.
How do I tell the real ClaudeBot from a fake?
Anthropic publishes the IP addresses its crawlers use at claude.com/crawling/bots.json and says a crawler on that list is coming from Anthropic. The list covers Anthropic’s crawlers as a group, not each bot separately. A request calling itself ClaudeBot from any other address is not Anthropic.
Should you block ClaudeBot?
It is the same trade-off as with GPTBot: blocking protects content you sell from being used in training; allowing it helps future Claude models know your brand and products. Whatever you decide for training, keep Claude-SearchBot allowed if you want Claude to cite you. Check your rules with the AI crawler checker.
Sources
Common questions
What is ClaudeBot's full user-agent string?
Anthropic's help article names the bot but does not publish a full user-agent string, so we don't print one here. Match requests on the token ClaudeBot and confirm the IP address against Anthropic's published list.
Can I slow ClaudeBot down instead of blocking it?
Yes. Anthropic says it supports the non-standard Crawl-delay extension to robots.txt, for example Crawl-delay: 1 in a User-agent: ClaudeBot group.
Is blocking ClaudeBot's IP addresses enough?
Anthropic says blocking IP addresses may not work correctly or persistently guarantee an opt-out, because it stops Anthropic from reading your robots.txt. Its documented opt-out is the robots.txt rule.