Skip to content
SEO Rank Fixer
Sign inStart free

Robots.txt Tester: Is This URL Allowed or Blocked?

Test a robots.txt: pick a crawler like Googlebot or GPTBot and a URL, and see if it is allowed or blocked and which rule decides, using Google's rules.

Updated

AllowedGooglebot → /admin/help

Rules used: the User-agent: * group (there is no group naming Googlebot). Deciding rule, line 3: Allow: /admin/help (the longest matching rule wins; on a tie, Allow wins).

   1  User-agent: *   2  Disallow: /admin/   3  Allow: /admin/help   4     5  User-agent: GPTBot
This path for every common crawler

Matching follows Google's rules (RFC 9309): a crawler obeys the group that names it (case does not matter) and falls back to User-agent: * only when none does; * matches any characters and $ marks the end of the URL. Some crawlers differ in edge cases (Google's image crawler, for one, also falls back to a Googlebot group).

A bad robots.txt edit can hide a whole site. SEO Rank Fixer checks yours every day and alerts you when a rule starts blocking pages that get clicks. Start free

How this robots.txt tester decides

Load your live file or paste a draft, pick a crawler and type a path or full URL. The tester shows Allowed or Blocked as you type, the group of rules that applied, and the exact line that decided it. It follows Google's documented rules, which the RFC 9309 robots.txt standard also sets out:

  • One group per crawler. A crawler uses the group whose User-agent line names it (case does not matter), and falls back to User-agent: * only when no group names it. Several groups naming the same crawler are merged.
  • Longest match wins. Among the rules that match the path, the longest one decides; on a tie, Allow beats Disallow. Line order is irrelevant.
  • Wildcards. * matches any characters and $ anchors the end of the URL. Matching is case-sensitive for paths and includes the query string.
  • Empty Disallow. Disallow: with nothing after it allows everything.

Nothing you paste leaves your browser: the matching runs on the page. Only "Fetch robots.txt" asks our server to download the file for you.

Common robots.txt mistakes

  • A leftover Disallow: / from a staging site, which blocks every crawler from the whole site.
  • Blocking CSS and JavaScript folders. Google renders pages like a browser; if it cannot load your styles and scripts it may misjudge the page.
  • Using robots.txt to hide private pages. The file is public and lists exactly what you want hidden. Use passwords or noindex instead.
  • A crawler-specific group without the shared rules. Adding a User-agent: GPTBot group to allow it also drops every * rule for it.

To see which AI crawlers your live robots.txt and firewall let in, run the AI crawler checker. For a whole-page check, try the SEO and AEO checker.

Common questions

Does Disallow in robots.txt remove a page from Google?

No. Disallow stops Google from crawling the page, but a blocked page can still be indexed from links to it, usually shown without a description. To keep a page out of results, allow crawling and add a noindex meta tag or X-Robots-Tag header; Google has to crawl the page to see it.

Which rule wins when Allow and Disallow both match?

Under Google's rules (also in the RFC 9309 standard), the most specific rule wins: the one with the longest path. If an Allow and a Disallow are equally long, Allow wins. The order of the lines does not matter.

Why does Googlebot ignore my User-agent: * rules?

A crawler follows only one group: the one that names it. If your file has a 'User-agent: Googlebot' group, Googlebot ignores the '*' group completely, so any Disallow lines you want for it must be repeated in its own group.

What do * and $ mean in robots.txt?

* matches any run of characters, so 'Disallow: /*?sort=' blocks every URL with a sort parameter. $ marks the end of the URL, so 'Disallow: /*.pdf$' blocks addresses ending in .pdf but not /file.pdf?download=1.

What happens if my robots.txt returns an error?

Google treats a 4xx answer (such as 404) as if there were no robots.txt: everything may be crawled. A 5xx server error makes Google stop crawling the site for a while, since it cannot tell what is allowed. Keep robots.txt fast and reachable.