Web crawling and crawlability

1Overview

Google describes Search as three stages: crawling, where it downloads text, images and video from pages it has found; indexing, where it analyses that content; and serving, where it returns results for queries [1]. Not every page that is crawled gets indexed [1].

A crawler starts from URLs it already knows, fetches each one, extracts the links it contains and adds new URLs to a queue. Google uses algorithms to decide which sites to crawl, how often and how many pages to fetch from each [1].

2History

The Robots Exclusion Protocol was proposed in 1994 and became the de facto way for site owners to tell crawlers which paths to avoid, but it remained an informal convention for 25 years [6].

In July 2019 Google proposed formalising the protocol as an internet standard and open-sourced the parser it uses to read robots.txt files [6]. The same year Googlebot moved to an “evergreen” version of Chromium, so it renders pages with a current browser engine [14]. The standard was published as RFC 9309 in September 2022 [5].

3Google’s crawlers

Google runs several crawlers. Common crawlers such as Googlebot obey robots.txt; special-case crawlers serve specific products; and user-triggered fetchers act on a user’s request and may ignore robots.txt [2].

Googlebot has two main types, Googlebot Smartphone and Googlebot Desktop. Because Google uses mobile-first indexing, most pages are crawled primarily with the smartphone agent [3].

User-agent strings can be faked. Google recommends verifying a crawler with a reverse DNS lookup to a googlebot.com or google.com hostname, followed by a forward lookup, or by matching published IP ranges [4].

4URL discovery

Most new URLs are found by following links from pages Google already knows. Others come from sitemaps submitted by site owners [1][16].

Google can only reliably follow links written as <a> elements with an href attribute that resolves to a URL. Links that exist only as JavaScript event handlers may not be discovered [15].

Some other search engines, including Bing, also accept instant URL notifications through the IndexNow protocol [19]. Google does not list IndexNow among its supported methods; it relies on links and sitemaps [1][16].

5robots.txt

A robots.txt file sits at the root of a host and tells crawlers which paths they may fetch. Google says it is mainly for managing crawler traffic, and is not a mechanism for keeping pages out of Google [7].

When several rules match a URL, Google applies the most specific, the longest matching path, and uses allow when an allow and disallow rule are equally specific [8]. Paths are case-sensitive [8].

How failures are handled

6Crawl budget

Google defines crawl budget as the set of URLs it can and wants to crawl on a site, shaped by two things: the crawl capacity limit, based on how much load the server can handle, and crawl demand, based on how popular and how fresh the content is [9].

Its guidance is written for large or fast-changing sites, roughly a million or more unique pages changing weekly, or ten thousand or more changing daily, and for sites with many pages reported as discovered but not indexed. Most sites don’t need to think about it [9].

• Consolidate duplicate URLs and block low-value infinite spaces such as faceted filters [9][17].
• Return 404 or 410 for permanently removed pages [9].
• Keep sitemaps current and avoid long redirect chains [9].
• Watch response times: a faster, more reliable server raises the crawl capacity limit [9].

Search Console’s Crawl Stats report shows Googlebot’s requests, response codes, file types and average response time for a site [10].

7Controlling crawl rate

To slow Googlebot down urgently, Google recommends temporarily returning 500, 503 or 429 status codes; prolonged errors, though, can lead to URLs being dropped [11][18].

Search Console previously offered a crawl rate limiter setting. Google announced its removal in November 2023, saying its crawling now adjusts to server load automatically [12].

8Rendering JavaScript

Google processes JavaScript pages in three phases: crawling, rendering and indexing. Pages may wait in a queue before rendering, which runs the page in a headless, evergreen Chromium [13][14].

Content and links that only appear after JavaScript runs can be indexed, but they depend on rendering succeeding. Server-side or pre-rendered HTML makes discovery faster and more reliable [13].

9Common crawl problems

10Common misconceptions