Search indexing, noindex and canonical URLs

1Overview

After crawling a page, Google processes its text, tags and attributes, and media to understand what it is about. During this stage it also decides whether the page is a duplicate of another and which version is canonical. The result may be stored in the Google index, a large database spread across many computers [1].

Indexing is not guaranteed. Google says not every page it processes will be indexed, and that whether a page is indexed depends on its content and metadata [1].

To be eligible at all, a page must meet Google’s technical requirements: Googlebot isn’t blocked, the page works and returns HTTP 200, and it has indexable content [17].

2Status codes

The HTTP status code a URL returns sets an upper limit on whether it can be indexed.

3The noindex directive

A noindex rule tells search engines not to include a page in their results. It can be set in an HTML robots meta tag or in an X-Robots-Tag HTTP response header; the header form also works for non-HTML files such as PDFs and images [2][3].

<meta name="robots" content="noindex"> applies to all crawlers that support it, while <meta name="googlebot" content="noindex"> targets Google only [3].

When Googlebot next crawls the page and sees the rule, Google drops it from results, regardless of whether other sites link to it [2].

noindex and robots.txt

robots.txt controls crawling, not indexing. A URL disallowed in robots.txt can still be indexed if linked from elsewhere, usually shown without a description [4]. In July 2019 Google announced it would stop supporting noindex rules written inside robots.txt, which had never been part of the protocol [13].

4Canonicalization

When several URLs show the same or very similar content, Google groups them and selects one as canonical: the version it crawls most regularly and shows in results. The others are treated as duplicates [5].

Duplicates arise for many reasons: http and https versions, www and non-www hosts, tracking parameters, sorting and filtering options, separate mobile URLs, and printer-friendly pages [5].

Signals Google uses

Google introduced support for the rel="canonical" link element in February 2009 [8]. It was later standardised by the IETF as RFC 6596 in 2012 [7].

5Canonical best practice

• Use absolute URLs, and point to a page that returns 200, not a redirect or an error [6].
• Don’t use robots.txt for canonicalisation, and don’t use the URL removal tool to choose canonicals [6].
• Don’t specify different canonicals for the same page through different methods [6].
• In hreflang clusters, each language version should self-canonicalise, not point to another language [15].
• Keep internal links, sitemaps and canonicals pointing at the same URL [6].

6Mobile-first indexing

Google predominantly uses the mobile version of a page’s content, crawled with the smartphone agent, for indexing and ranking [14].

Content, structured data, metadata and directives such as noindex should therefore be the same on mobile and desktop. Material present only on desktop may not be indexed [14].

7Diagnosing indexing

Search Console’s Page indexing report shows how many of a site’s URLs are indexed and groups excluded URLs by reason, such as “Excluded by ‘noindex’ tag”, “Alternate page with proper canonical tag” or “Crawled: currently not indexed” [10].

The URL Inspection tool shows Google’s indexed version of a single URL, including the user-declared canonical and the Google-selected canonical, and can test the live page [11].

For urgent cases, the Removals tool hides a URL from results for about six months while a permanent fix such as noindex or deletion takes effect [12].

8Common misconceptions