GPTBot: OpenAI’s training crawler, and how to block it
GPTBot collects pages to train OpenAI’s models. Its user agent, IP ranges, robots.txt rules, and why blocking it does not affect ChatGPT search.
Updated
GPTBot is OpenAI’s web crawler for model training. OpenAI says it is used to make its generative AI foundation models more useful and safe, and that it crawls content that may be used in training them. Blocking it with User-agent: GPTBot in robots.txt opts your site out of that training, without affecting whether ChatGPT search can show your pages.
| Owner | OpenAI |
|---|---|
| Purpose | Training |
| robots.txt token | GPTBot |
| Follows robots.txt | Yes |
| Published IP ranges | openai.com/gptbot.json |
| If you block it | Your pages are not used to train OpenAI’s models. ChatGPT search is unaffected. |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbotOpenAI notes that the version number in the string may change, and that when GPTBot fetches robots.txt itself it may add a robots.txt marker to the string. Match on the token GPTBot, not the full string.
What does GPTBot do?
It crawls public pages to build training data for OpenAI’s foundation models. It is not the crawler that powers ChatGPT’s search answers; that is OAI-SearchBot. And it is not the agent ChatGPT uses when a user asks it to open a page; that is ChatGPT-User. OpenAI says that if you allow both GPTBot and OAI-SearchBot, it may use the results from one crawl for both purposes to avoid duplicate crawling.
What happens if I block GPTBot?
Your pages are signaled as off-limits for training OpenAI’s future models. Your visibility in ChatGPT search stays under the control of OAI-SearchBot. What you give up is harder to measure: what a future model “knows” about your brand without searching comes partly from training data, so a blocked site may be described less accurately from memory. That trade-off is a business decision; there is no right answer for every site.
How do I block or allow GPTBot?
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /User-agent: GPTBot
Disallow: /members/
Disallow: /premium/User-agent: GPTBot
Allow: /Remember that a crawler follows only the most specific group that names it. If you add a User-agent: GPTBot group, GPTBot ignores your User-agent: * rules, so repeat any paths you still want hidden.
How do I tell the real GPTBot from a fake?
Anyone can send a request claiming to be GPTBot. Check the request’s IP address against the list OpenAI publishes at openai.com/gptbot.json. A request with the GPTBot user agent from an IP outside that list is not OpenAI’s crawler, and you can block it at your firewall without affecting OpenAI.
Should you block GPTBot?
Block it if you sell content (news, research, courses) and don’t want it used for training without a deal. Allow it if your goal is to be well known to AI assistants and your pages are marketing material you want widely repeated. Either way, keep OAI-SearchBot allowed if you want ChatGPT to cite you. Check your current rules with the AI crawler checker.
Sources
Common questions
Does blocking GPTBot remove my site from ChatGPT?
No. OpenAI uses a separate crawler, OAI-SearchBot, for ChatGPT search, and says each robots.txt setting is independent. You can block GPTBot to opt out of training and still allow OAI-SearchBot so ChatGPT can cite you.
Does blocking GPTBot remove content OpenAI already collected?
OpenAI's documentation describes the robots.txt setting as a signal about crawling and use from now on. It does not describe removing content collected before you added the rule.
How long until OpenAI sees my robots.txt change?
OpenAI says it can take about 24 hours from a robots.txt update for its systems to adjust.