Skip to content
SEO Rank Fixer
Sign inStart free

What is robots.txt?

Robots.txt is a text file at your site's root that tells crawlers which URLs they may fetch. What it can and cannot do, with examples.

Updated

Robots.txt is a plain text file at the root of a site (for example https://example.com/robots.txt) that tells crawlers which URLs they may and may not fetch. It follows the Robots Exclusion Protocol, published as RFC 9309 in 2022. Well-behaved crawlers such as Googlebot, Bingbot and most AI search crawlers obey it; it is a request, not a lock.

What does a robots.txt file look like?

robots.txt
User-agent: *
Disallow: /cart/
Disallow: /search

User-agent: GPTBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

Each group starts with one or more User-agent lines naming crawlers, followed by Allow and Disallow rules. A crawler follows the most specific group that names it and ignores the rest, so in the example GPTBot is blocked from everything while other crawlers are only kept out of the cart and search pages.

What robots.txt cannot do

  • Keep a page out of Google. Google says robots.txt is not a mechanism for that: a blocked URL can still be indexed, without its content, if other pages link to it. Use noindex instead.
  • Hide private content. The file is public, and bad bots ignore it. Use authentication.
  • Control what AI does with pages it already has. It controls fetching from now on.

Common mistakes

  • A leftover Disallow: / from a staging site.
  • Blocking CSS or JavaScript files Google needs to render the page.
  • Blocking a page you also marked noindex, so the noindex is never seen.
  • Blocking AI search crawlers by accident while meaning to block only training crawlers (see the AI crawlers guide).

Test rules for any URL and user agent with the robots.txt tester, and see which AI bots you let in with the AI crawler checker.

Sources