XML sitemaps

1Overview

Search engines find most pages through links. A sitemap gives them a second route: a list of URLs the site owner considers important, which can include pages that are new, poorly linked or deep in the site [2].

Google says a sitemap is most useful for sites that are large, new with few external links, have many pages not well linked to each other, or have a lot of rich media or news content. A small site with good internal links may not need one [2].

2History

The Sitemaps protocol began as a Google format and was published at sitemaps.org as version 0.90 under a Creative Commons licence, after which it was adopted by other major search engines [1]. Later additions included the Sitemap: line in robots.txt for automatic discovery [1].

In June 2023 Google retired its sitemap “ping” endpoint, saying most pings it received were not useful, and confirmed it would use <lastmod> only when it is consistently accurate [5].

3Supported formats

4XML structure

An XML sitemap has a <urlset> root element with the sitemaps.org namespace, and one <url> block per page [1].

Files must be UTF-8 encoded, and characters such as & in URLs must be entity-escaped [1].

5Size limits and index files

Each sitemap file may contain at most 50,000 URLs and be no larger than 50 MB uncompressed; files may be gzip-compressed for transfer [1][3].

Larger sites split their URLs across several sitemaps and list them in a sitemap index file, which uses a <sitemapindex> root and the same limits. The index can be submitted once instead of each file [1][4].

Splitting by content type, products, categories, articles, has a practical benefit: reports per file show which part of the site has indexing problems.

6Location and submission

Google recommends placing the sitemap at the site root. A sitemap generally affects only URLs in the directory it is placed in and below [3].

• Submit it in Search Console’s Sitemaps report, which shows when it was last read and any errors [6].
• Add a Sitemap: line to robots.txt so any crawler can find it; the line can appear more than once and is independent of user-agent groups [11].
• Since the 2023 ping retirement, Google relies on these two methods and on recrawling known sitemaps [5].

7Extensions

Google supports sitemap extensions that add information about specific content types [3]:

• Image: lists images on each page, helping Google find images it might not otherwise discover [7].
• Video: describes video content such as title, thumbnail and duration [8].
• News, for publishers in Google News, listing recently published articles [9].
• hreflang: declares localised alternates for each URL, as an alternative to HTML tags [10].

8What to include

Google asks for the full, absolute URLs of the pages you want shown in search: the canonical versions [3]. Listing a URL in a sitemap is a weak canonicalisation signal, so it should agree with redirects and rel="canonical"[12].

9Common misconceptions