What is crawl budget?

Crawl budget is the number of URLs Google can and wants to crawl on a site in a given period. Google defines it as the combination of crawl capacity (how much crawling your server can take without slowing down) and crawl demand (how much Google wants to crawl, based on how popular and how often updated your pages are).

Do I need to worry about crawl budget?

Probably not. Google’s own guide is written for very large or fast-changing sites, giving rough examples of sites with over a million unique pages changing about weekly, or over 10,000 pages changing daily. If most new pages on your site are crawled within a day or so of publishing, crawl budget is not your problem.

For a smaller site, pages that are Discovered - currently not indexed usually point to weak internal links or low perceived value, not a lack of budget.

What wastes crawl budget on large sites?

• Endless URL variations from filters, sorting and session parameters.
• Long redirect chains and soft 404 pages.
• Duplicate content at many URLs.
• Slow server responses, and 5xx or 429 errors, which make Google slow its crawl rate down.

How do you improve it?

• Block crawl traps (infinite filters, internal search) in robots.txt.
• Keep an accurate sitemap with true lastmod dates.
• Return 404 or 410 for pages that are gone, and fix redirect chains.
• Make the server faster.

Example

An online store with 5,000 products and faceted navigation can expose millions of crawlable filter URLs. Google spends its visits on ?color=red&size=9&sort=price combinations while new products wait. Blocking those parameters lets the crawl reach the products.

Search Console’s Crawl stats report (Settings, then Crawl stats) shows how many requests Google makes and how fast your server answers.