Claude-SearchBot: the crawler behind Claude’s search
Claude-SearchBot indexes pages so Claude can find and cite them in search answers. What blocking it costs and how to allow it in robots.txt.
Updated
Claude-SearchBot is the Anthropic crawler that navigates the web to improve the quality of search results for Claude users. Anthropic says disabling it prevents its system from indexing your content for search optimization, which may reduce your site’s visibility and accuracy in user search results. If you want Claude to find and cite your pages when it searches, allow it.
| Owner | Anthropic |
|---|---|
| Purpose | Search |
| robots.txt token | Claude-SearchBot |
| Follows robots.txt | Yes |
| Published IP ranges | claude.com/crawling/bots.json |
| If you block it | Your pages are not indexed for Claude’s search, which may reduce visibility in its answers. |
What does Claude-SearchBot do?
Anthropic describes it as analyzing online content specifically to enhance the relevance and accuracy of search responses. In practice it is the search-side counterpart of ClaudeBot: one is about training, the other about what Claude can find when it searches the web to answer a question.
What happens if I block Claude-SearchBot?
Anthropic’s own wording is that it may reduce your site’s visibility and accuracy in user search results. Claude may still know of you from training or from other sites, but it is less likely to have your current pages to quote and link. For a business that wants to be recommended, that is a cost with little upside.
How do I allow it?
User-agent: Claude-SearchBot
Allow: /If your robots.txt has a User-agent: * group with broad Disallow rules, Claude-SearchBot follows that group unless you give it its own, so add an explicit group like the one above.
How do I block it?
User-agent: Claude-SearchBot
Disallow: /How do I verify it?
Check the IP against Anthropic’s published crawler list at claude.com/crawling/bots.json. Anthropic does not label which address belongs to which bot.
Should you allow Claude-SearchBot?
If you want visibility in Claude’s answers, yes. A common setup is to block ClaudeBot (training) and allow Claude-SearchBot and Claude-User. Test your file with the AI crawler checker. SEO Rank Fixer’s AI visibility check shows whether Claude actually cites or names you for your customers’ questions.
Sources
Common questions
Is Claude-SearchBot the same as ClaudeBot?
No. Anthropic lists them as separate bots. ClaudeBot collects content that may be used for training; Claude-SearchBot analyzes content to improve the relevance and accuracy of search responses. Each is controlled by its own robots.txt group.
Will allowing Claude-SearchBot make Claude cite me?
It makes your pages available to Claude's search. Whether Claude cites you depends on whether your page answers the question well, like any search system.
What is its user-agent string?
Anthropic's help article names Claude-SearchBot but does not publish a full user-agent string. Match on the token and verify the IP against Anthropic's list.