Claude-User: when Claude fetches a page for a user
Claude-User opens pages when someone asks Claude about them. What it does, what blocking it changes, and the robots.txt rules to control it.
Updated
Claude-User is the agent Claude uses to visit a website when a person asks Claude a question that needs it. Anthropic says disabling Claude-User prevents its system from retrieving your content in response to a user query, which may reduce your site’s visibility for user-directed web search. It is controlled with its own User-agent: Claude-User group in robots.txt.
| Owner | Anthropic |
|---|---|
| Purpose | User-triggered |
| robots.txt token | Claude-User |
| Follows robots.txt | Yes |
| Published IP ranges | claude.com/crawling/bots.json |
| If you block it | Claude can’t fetch your pages for users who ask about them. |
What does Claude-User do?
When someone asks Claude about your product, pastes your URL, or asks a question Claude decides to research, Claude may open pages on your site using the Claude-User agent. Each visit is tied to a person’s request, not to a crawl schedule. Anthropic’s other agents are ClaudeBot (training) and Claude-SearchBot (search indexing).
What happens if I block it?
Claude can’t read your pages while helping that user, so it has to rely on what other sites say about you or on what it learned in training, which may be out of date. For a business, those users are often prospects in the middle of a decision.
How do I control Claude-User in robots.txt?
User-agent: Claude-User
Disallow: /account/
Disallow: /admin/User-agent: Claude-User
Disallow: /How do I verify it?
Anthropic doesn’t publish a full user-agent string for Claude-User in its help article. Match the token, then check the request IP against claude.com/crawling/bots.json, Anthropic’s list of crawler addresses.
Should you allow Claude-User?
For public marketing pages, documentation and product pages, yes: these are the visits where accurate information helps you most. Keep private content behind a login rather than relying on robots.txt. Check all of your AI bot rules at once with the AI crawler checker.
Sources
Common questions
Does Claude-User follow robots.txt?
Anthropic's help article says its bots honor industry-standard robots.txt directives, and it describes Claude-User as letting site owners control which sites can be accessed through user-initiated requests. Unlike OpenAI and Perplexity, it does not carve out an exception for user-triggered fetches.
Does blocking Claude-User stop Claude from training on my site?
No. Training is controlled by ClaudeBot. Claude-User only covers pages fetched to answer a user's question.