OAI-SearchBot: the crawler behind ChatGPT search
OAI-SearchBot decides whether your site can appear in ChatGPT search answers. Its user agent, IP ranges, and how to allow it in robots.txt.
Updated
OAI-SearchBot is the OpenAI crawler that finds and indexes pages for ChatGPT’s search features. OpenAI says sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. If you want ChatGPT to cite and link your pages, this is the crawler to allow.
| Owner | OpenAI |
|---|---|
| Purpose | Search |
| robots.txt token | OAI-SearchBot |
| Follows robots.txt | Yes |
| Published IP ranges | openai.com/searchbot.json |
| If you block it | You are not shown in ChatGPT search answers (navigational links may remain). |
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbotWhen it fetches robots.txt, OpenAI says OAI-SearchBot may add robots.txt; to the string before the URL. The version number can change, so match on OAI-SearchBot.
What does OAI-SearchBot do?
It crawls the web so ChatGPT can surface sites in its search answers, the answers with source links that appear when ChatGPT searches. OpenAI recommends allowing OAI-SearchBot in robots.txt, and allowing requests from its published IP ranges, to help your site appear in search results. It is separate from GPTBot (training) and ChatGPT-User (fetches a user asked for).
What happens if I block OAI-SearchBot?
Your pages won’t be shown in ChatGPT search answers. For most businesses that is a real cost: people increasingly ask ChatGPT for recommendations and comparisons, and a blocked site can’t be one of the cited sources. Many sites block it by accident, with a blanket rule meant for AI training bots, or through a CDN’s “block AI bots” switch.
How do I allow OAI-SearchBot?
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /User-agent: OAI-SearchBot
Disallow: /account/
Disallow: /checkout/OpenAI says changes can take about 24 hours to be reflected in search results.
How do I block it?
User-agent: OAI-SearchBot
Disallow: /How do I tell the real OAI-SearchBot from a fake?
Compare the request IP with OpenAI’s list at openai.com/searchbot.json. If you allowlist bots at a firewall, use these ranges together with the user agent, as OpenAI suggests, rather than the user agent alone.
Should you allow OAI-SearchBot?
For almost every business, marketing site and publisher that wants traffic: yes. It is the switch for being citable in ChatGPT search. Check that both robots.txt and your CDN let it through with the AI crawler checker, and read how to get recommended by ChatGPT.
Sources
Common questions
Is OAI-SearchBot used to train OpenAI's models?
OpenAI describes OAI-SearchBot as the crawler used to surface websites in ChatGPT's search features, separate from GPTBot, which collects training data. OpenAI notes that if both are allowed it may use one crawl's results for both purposes, so block GPTBot if you want to opt out of training.
I allowed OAI-SearchBot. Why doesn't ChatGPT cite me?
Being crawlable makes you eligible, not chosen. ChatGPT cites the pages that best answer the specific question. Clear answers, specific facts and being described on other sites matter, just as in search.
Does a firewall or CDN block it even if robots.txt allows it?
It can. Bot protection at a CDN or firewall can challenge or block crawlers regardless of robots.txt. OpenAI recommends allowing requests from its published IP ranges.