Before a page can rank, a search engine has to reach it. Crawlability covers robots.txt, crawl budget, and the structural issues that stop bots from discovering your content. Get this wrong and the rest of your SEO never gets a chance.
Crawling, then indexing, then ranking. They happen in that order, and each depends on the one before it. A page that can't be crawled can't be indexed, and a page that isn't indexed can't rank.
Most crawlability problems come from one of three places: a robots.txt rule that blocks more than intended, a structure where important pages have no internal links pointing to them, or crawl traps (redirect chains, infinite parameter URLs) that waste a bot's time. The explainers below cover the two areas you most need to understand: how robots.txt actually behaves, and when crawl budget is worth worrying about.
The two things people most often get wrong about how crawlers reach pages.
How robots.txt works, the difference between blocking crawling and blocking indexing, and the over-broad Disallow rules that accidentally hide whole sections from search.
What crawl budget is, when it actually matters (and when it doesn't), and how redirects, parameters, and low-value URLs quietly burn it on large sites.
Grounded in the real checks the crawler runs.
robots.txt missing · robots.txt blocks the sitemap · robots.txt blocks indexable pages · robots.txt doesn't declare a sitemap · pages blocked from crawling · redirect chains and loops that trap crawlers · orphan pages with no path for discovery · deep pages more than four clicks from the homepage. Crawlability overlaps with the Links, Redirects and Sitemaps categories, because discovery depends on all three.
Free to start. Find blocked pages, crawl traps and discovery gaps across your site.
Start my free audit