Crawl budget is how much of your site search engines will crawl in a given window. For small sites it barely matters; for large ones, wasted crawl budget means important pages get crawled and refreshed less often. Here's crawl budget explained, and how to protect it.
Two forces decide it: how much your server can handle, and how much Google wants to crawl you. Where those meet is your budget.
Search engines don't crawl every URL on the web constantly, they ration. Crawl capacity is how many requests your server can take without slowing down; a fast, reliable server earns more crawling, a slow or error-prone one earns less. Crawl demand is how much Google wants to crawl you, driven by your site's size, popularity, and how often your content actually changes. The practical budget is where capacity and demand meet, and the question that matters is whether that budget covers all the URLs worth crawling.
Be honest about which situation you're in before spending time on it.
If your site has up to a few thousand URLs, Google can crawl all of it easily and frequently. Worrying about crawl budget here is a distraction: your time is far better spent on content quality and internal links. Don't optimise a problem you don't have.
Crawl budget becomes real when you have tens of thousands of URLs, or when your site generates many low-value URLs automatically: faceted navigation, filter and sort parameters, session IDs, infinite calendars, or near-duplicate pages. On these sites, Google can spend its budget on the noise and crawl your important pages rarely, so new content is indexed slowly and updates take a long time to register.
Most crawl-budget problems are the same technical issues this guide already covers, they just compound at scale.
Broken links and 404s send crawlers to dead ends they then re-check, see why broken links hurt SEO. Redirect chains make Google follow multiple hops for one page, see redirect chains. Parameter and faceted URLs multiply near-duplicate pages; consolidate with canonicals or block low-value parameters in robots.txt. Duplicate content means crawling the same thing many times. A bloated sitemap full of non-indexable URLs points crawlers at the wrong pages. A slow server directly lowers crawl capacity. Fix these and the available budget flows to your real, indexable content, which is the entire goal.
Two Search Console reports answer this: Crawl Stats and the Pages report. The signs of a genuine problem are a large number of URLs sitting in "Discovered: currently not indexed" or "Crawled: currently not indexed", important pages that are crawled infrequently, and a wide gap between how many URLs your site has and how many are indexed.
If none of those appear, and your small site is fully indexed, crawl budget is not what is holding you back. Diagnose before you optimise: time spent trimming crawl waste on a site Google already covers completely is time taken from work that would actually move rankings.
There is no direct ranking effect. What crawl budget governs is discovery and freshness, and both feed rankings indirectly. A page that is not crawled cannot be indexed, and a page that is crawled rarely cannot have its updates reflected in the index promptly.
So the goal of crawl-budget work is to make sure your good pages get crawled and re-crawled, not to earn a ranking boost from crawling itself. Framing it that way keeps expectations honest and points the effort at the right pages.
Blocking genuinely low-value paths, internal search results and infinite parameter URLs being the classic examples, can help push crawling toward the pages that deserve it. That is a legitimate use of robots.txt and one of the few crawl-budget levers with an immediate effect.
Two cautions apply. Be careful not to block pages you want indexed, since an over-broad rule is easy to write and hard to notice. And remember that robots.txt does not deindex anything: a blocked URL can stay in the index without a description. The robots.txt guide covers both traps.
Free to start. Find the broken links, redirect chains and duplicate URLs burning your crawl budget.
Start my free audit