Blocked by robots.txt in Search Console: how to fix

Blocked by robots.txt means a rule in your site’s robots.txt file tells Googlebot not to crawl this URL, so Google did not read it. Often that is deliberate. It becomes a problem when the blocked URL is a page you want in search, or when you used robots.txt hoping to hide a page, which it doesn’t reliably do.

What does it mean?

Google’s help page says the page was blocked by your site’s robots.txt file, and notes that this does not guarantee the page won’t be indexed through other means. Google never fetched the content, so it can’t judge it, see its canonical tag or read a noindex rule on it.

A typical rule that causes it:

User-agent: *
Disallow: /blog/drafts/
Disallow: /search

Is it a problem?

Blocking crawl traps, cart and account pages, and internal search is good practice. It is a problem when the blocked list includes product, service or article pages, or CSS and JavaScript files Google needs to render your pages. Also check that the blocked URLs are not in your sitemap: listing a URL in the sitemap while blocking it sends opposite instructions.

What causes it, most likely first?

1. Intentional blocks on utility URLs, which need no fix.
2. Rules that match more than intended, such as Disallow: /shop also blocking /shop-guide/, or wildcard patterns.
3. A staging robots.txt with Disallow: / pushed to the live site.
4. CMS or plugin defaults that block folders or parameters.
5. Using robots.txt to “noindex” pages, which Google does not support.

How do I fix it?

1. 1. Find the rule that blocks the URL

   Open your robots.txt (yoursite.com/robots.txt) and test the blocked URL against it for the Googlebot user agent with a robots.txt tester. Note which Disallow line matches. Google uses the most specific matching rule, and Allow wins a tie.

2. 2. Decide whether the page should be crawled

   If it is a cart, account, internal search or filter URL, the block is probably right; take the URL out of your sitemap and stop. If it is a page you want in Google, continue.

3. 3. Narrow or remove the rule

   Edit robots.txt so the page is allowed: delete the Disallow line, make its path more specific, or add an Allow line for the section you want crawled. Watch out for broad patterns such as Disallow: /p that also match /products/.

4. 4. Check robots.txt in Search Console

   Open Settings, then robots.txt, to see the version Google last fetched and any fetch errors. You can ask Google to recrawl the file there after an urgent fix.

5. 5. If you wanted the page out of Google, use noindex instead

   Robots.txt does not reliably keep a page out of search. To remove a page, allow crawling and add a noindex meta tag or X-Robots-Tag header, so Google can see the rule.

6. 6. Request indexing and validate

   Request indexing for important unblocked pages in URL Inspection, then click Validate fix on the reason in the Page indexing report.

How do I check that the fix worked?

Test the URL against the updated file with the robots.txt tester. In URL Inspection, Test live URL should report Crawl allowed? Yes. After recrawling, the page moves to indexed or to another reason. Validate fix on the reason tracks the group.

How long does it take?

Google usually picks up a changed robots.txt within a day. The unblocked pages then need to be crawled and processed, which takes days for requested pages and longer for large sections.

Learn the rules in our robots.txt entry, and see which AI crawlers your file lets in with the AI crawler checker.