Before a page can rank, a search engine has to reach it. Crawlability covers robots.txt, crawl budget, and the structural issues that stop bots from discovering your content. Get this wrong and the rest of your SEO never gets a chance.
Crawling, then indexing, then ranking. They happen in that order, and each depends on the one before it. A page that can't be crawled can't be indexed, and a page that isn't indexed can't rank.
Search engines work in a strict sequence: crawl, then index, then rank. Each step depends entirely on the one before it. This is why crawlability is the very first thing to get right. A brilliant page with perfect content and strong links is worth exactly nothing in search if Googlebot can't reach it in the first place, because the chain never even starts.
Most crawlability problems trace back to one of three causes, and the explainers below go deep on the two that catch people most often.
An over-broad Disallow rule, often a Disallow: / left over from staging, stops crawlers reaching pages you actually want indexed. The most common self-inflicted crawl wound.
Pages with no internal links pointing to them (orphans) sit outside the crawl. Bots find pages by following links; if nothing links to a page, it may never be found. Orphan pages →
Redirect chains, infinite calendars, and endless parameter URLs waste a bot's limited time on junk instead of your real content, slowing how fast everything else is found. Redirect chains →
Search works in one direction. A page can only reach a stage if it cleared the one before it, which is why crawlability comes first.
A bot fetches the page. Blocked by robots.txt or undiscoverable? It stops right here, and nothing downstream can happen.
The crawled page is analysed and stored. A noindex tag or a bad canonical removes it at this stage. Indexability →
Only indexed pages compete in results. This is where content, links and performance finally pay off.
The two things people most often get wrong about how crawlers reach pages.
How robots.txt works, the difference between blocking crawling and blocking indexing, and the over-broad Disallow rules that accidentally hide whole sections from search.
What crawl budget is, when it actually matters (and when it doesn't), and how redirects, parameters, and low-value URLs quietly burn it on large sites.
Grounded in the real checks the crawler runs.
Crawlability overlaps with the Links, Redirects and Sitemaps categories, because discovery depends on all three.
Crawlability is whether search engine bots can reach and read your pages. It depends on your robots.txt rules, your internal link structure (a page with no links to it can't be discovered), a reliable server, and the absence of crawl traps like infinite redirect chains. If a page isn't crawlable it can't be indexed, so crawlability is the very first thing to get right.
No, and this is a costly misconception. Blocking a URL in robots.txt stops crawling but does not remove it from the index, and it prevents Google from seeing a noindex tag you add later (it can't crawl the page to read it). To remove a page, allow crawling and add a noindex. Use robots.txt to manage crawling, not indexing.
Crawl budget is how many URLs a search engine will crawl on your site in a given period. For small sites (a few thousand URLs) it rarely matters. It matters for large sites, sites with many low-value parameter/faceted URLs, or sites wasting crawls on redirects and 404s, where important pages then get crawled and refreshed less often. See crawl budget explained.
Use Search Console's URL Inspection tool for a specific page (it shows last crawl date and status) and the Crawl Stats report for site-wide patterns. Pages stuck in "Discovered, currently not indexed" are often a discovery or crawl-budget signal worth investigating.
Free to start. Find blocked pages, crawl traps and discovery gaps across your site.
Start my free audit