Crawl capacity limit and crawl demand
A URL gets crawled when the server has capacity and Google wants the page. Google calls these two factors the crawl capacity limit and crawl demand.
Crawl capacity limit
The crawl capacity limit (also called hostload) caps the total time a server spends holding connections open for Google.
Google treats 5xx responses and HTTP 429 as signals to temporarily slow crawling, which lowers how much of a site gets fetched.
All Google crawlers share one crawl capacity limit, so high demand from one crawler can leave less capacity for others.
Crawl demand
For Googlebot, crawl demand depends on a site's size, update frequency, page quality and relevance compared to other sites. Perceived inventory (how many known URLs Google tries to crawl, including duplicates) is the demand factor a site owner controls most. That makes duplicate URLs the first place to look.
Which sites need to manage it, and what wastes it
Google's crawl budget guide targets large sites, such as 1 million+ unique pages that change about weekly; Google calls its thresholds rough estimates. Google says sites with fewer than a thousand pages should not need the Crawl Stats report.
Two things Google tells you to clean up:
Soft 404s. Google's crawl-budget guide says soft 404 pages keep getting crawled and waste crawl budget.
Redirect chains. Each hop of a server-side redirect chain shows up in Crawl Stats as one more crawl request.
Two common tips do not work as expected. Google advises against using noindex to save crawl budget, because Google still requests the page before dropping it. Blocking URLs in robots.txt to free budget for other pages does not work unless Google is already hitting the site's crawl capacity limit.
What Google says to do instead: use robots.txt only for pages or resources you never want Google to crawl. Its examples are infinite scrolling pages that duplicate linked pages, and differently sorted versions of the same page.
Google names two ways to get more crawl budget: add server resources when capacity is the limit, and improve content quality for the targeted Google product. Server response time and its effect on search are covered in page speed and SEO. For crawling and indexing on a Next.js stack, see the Next.js SEO guide.
Sources
- Optimize your crawl budgetGoogle · developers.google.com
- How HTTP status codes, and network and DNS errors affect Google SearchGoogle · developers.google.com
- Crawl Stats report - Search Console HelpGoogle · support.google.com