Before a page can rank, a search engine has to reach it. Crawlability for SEO covers robots.txt, crawl budget, and the structural issues that stop bots from discovering your content. Get this wrong and the rest of your SEO never gets a chance.
Crawling, then indexing, then ranking. They happen in that order, and each depends on the one before it. A page that can't be crawled can't be indexed, and a page that isn't indexed can't rank.
Search engines work in a strict sequence: crawl, then index, then rank. Each step depends entirely on the one before it. This is why crawlability is the very first thing to get right. A brilliant page with perfect content and strong links is worth exactly nothing in search if Googlebot can't reach it in the first place, because the chain never even starts.
Most crawlability problems trace back to one of three causes, and the explainers below go deep on the two that catch people most often.
An over-broad Disallow rule, often a Disallow: / left over from staging, stops crawlers reaching pages you actually want indexed. The most common self-inflicted crawl wound.
Pages with no internal links pointing to them (orphans) sit outside the crawl. Bots find pages by following links; if nothing links to a page, it may never be found. Orphan pages →
Redirect chains, infinite calendars, and endless parameter URLs waste a bot's limited time on junk instead of your real content, slowing how fast everything else is found. Redirect chains →
Search works in one direction. A page can only reach a stage if it cleared the one before it, which is why crawlability comes first.
A bot fetches the page. Blocked by robots.txt or undiscoverable? It stops right here, and nothing downstream can happen.
The crawled page is analysed and stored. A noindex tag or a bad canonical removes it at this stage. Indexability →
Only indexed pages compete in results. This is where content, links and performance finally pay off.
The two things people most often get wrong about how crawlers reach pages.
How robots.txt works, the difference between blocking crawling and blocking indexing, and the over-broad Disallow rules that accidentally hide whole sections from search.
What crawl budget is, when it actually matters (and when it doesn't), and how redirects, parameters, and low-value URLs quietly burn it on large sites.
Grounded in the real checks the crawler runs.
Crawlability overlaps with the Links, Redirects and Sitemaps categories, because discovery depends on all three.
Crawlability is whether search engine bots can reach and read your pages at all. It depends on four things working together: your robots.txt rules, your internal link structure, a reliable server, and the absence of crawl traps such as infinite redirect chains.
Internal links deserve particular attention here, because a page with no links pointing to it cannot be discovered by a crawler walking your site. If a page is not crawlable it cannot be indexed, and if it is not indexed it cannot rank, which is why crawlability is the very first thing to get right in any audit.
Blocking a URL in robots.txt stops crawling, but it does not remove that URL from the index. This is one of the costliest misconceptions in technical SEO, and it gets worse in combination: a robots.txt block also prevents Google from seeing a noindex tag you add later, because it cannot crawl the page to read it.
To remove a page from search, allow crawling and add a noindex directive. Keep the two jobs separate in your head: robots.txt manages crawling, noindex manages indexing, and using the first for the second reliably backfires. The robots.txt guide covers this in full.
Crawl budget is how many URLs a search engine will crawl on your site in a given period. For small sites of a few thousand URLs it rarely matters, because Google can comfortably cover everything.
It becomes a real constraint on large sites, on sites with many low-value parameter or faceted URLs, and on sites wasting crawls on redirects and 404s. In those cases the important pages get crawled and refreshed less often, so new content is indexed slowly and updates take a long time to register. See crawl budget explained for how to tell which situation you are in.
Search Console's URL Inspection tool answers the question for a specific page, showing the last crawl date and the status Google recorded. For site-wide patterns, the Crawl Stats report is the better view: it shows how crawling is distributed and whether it is trending up or down.
Pages stuck in the "Discovered, currently not indexed" state are worth investigating first. That status usually points to a discovery problem or a crawl-budget signal rather than a content problem, so it tells you where to look.
Free to start. Find blocked pages, crawl traps and discovery gaps across your site.
Start my free audit