Skip to content
SEO Rank Fixer
Sign inStart free

AI crawlers and bots

Who the AI crawlers are, what they read, and how to control them on your site.

AI crawlers compared

Training crawlers collect pages for model training. Search crawlers decide whether you can be cited in AI search answers. User-triggered fetchers open a page because someone asked. Blocking one type does not block the others. Facts are from each owner’s documentation.

AI crawlers compared
CrawlerOwnerPurposeFollows robots.txtIf you block it
GPTBotOpenAITrainingYesYour pages are not used to train OpenAI’s models. ChatGPT search is unaffected.
OAI-SearchBotOpenAISearchYesYou are not shown in ChatGPT search answers (navigational links may remain).
ChatGPT-UserOpenAIUser-triggeredMay not apply (user-initiated)ChatGPT may fail to open your pages when a user asks it to. Not used for search inclusion.
ClaudeBotAnthropicTrainingYes, including Crawl-delayYour future pages are excluded from Anthropic’s training data.
Claude-SearchBotAnthropicSearchYesYour pages are not indexed for Claude’s search, which may reduce visibility in its answers.
Claude-UserAnthropicUser-triggeredYesClaude can’t fetch your pages for users who ask about them.
PerplexityBotPerplexitySearchYes (managed by robots.txt)Your site may not appear or be linked in Perplexity’s search results.
Perplexity-UserPerplexityUser-triggeredGenerally ignores it (user-initiated)A robots.txt block generally has no effect; use a firewall rule.
Google-ExtendedGoogleTraining opt-out tokenIt is a robots.txt tokenContent isn’t used for Gemini training or grounding. Google Search, including AI Overviews, is unaffected.
Applebot-ExtendedAppleTraining opt-out tokenIt is a robots.txt tokenContent isn’t used to train Apple’s foundation models. Applebot still crawls for search.
bingbotMicrosoftSearch and AIYesYou drop out of Bing, and of the AI answers grounded in Bing’s index.
CCBotCommon CrawlTrainingYes, including Crawl-delayNew pages stay out of Common Crawl’s open archive, which many AI training sets draw on.
BytespiderByteDanceTrainingNo official statement foundUnclear: ByteDance publishes no documentation. Block at the firewall to be sure.
meta-externalagentMetaTrainingYes (controlled by robots.txt)Your pages are not crawled for Meta’s AI training or its direct content indexing.

Check which of these your robots.txt lets in with the AI crawler checker.