Find out exactly why a page can or can’t be indexed: noindex tags, robots.txt blocks, canonical conflicts and HTTP status, checked together.
Every one has to be open. The checker above tests all five in order on the URL you enter, stops at the first that is shut, and names it — free, and in a few seconds.
Those five are properties of one URL, which is why a single fetch settles them. Across a whole site another 68 checks apply that no single fetch can see, because they are relationships between URLs: which page duplicates which, what the sitemap claims, where a canonical eventually lands, what nothing links to.
Kalenux Disallow: /blog/ would stop it here✓ PASSA 404, 410 or 5xx stops it here✓ PASS<meta name="robots" content="noindex">✕ BLOCKEDNot reachedNOT REACHEDNot reachedNOT REACHEDGate 3 is shut, so gates 4 and 5 are never reached. One blocked gate is enough.
Which URL Google keeps when several carry the same content.
A sitemap that contradicts the pages it lists.
Every way a page can be told not to be stored.
Where a URL actually ends up, not where it claims to.
The file that decides what gets fetched at all.
Pages nothing links to, which Google reaches late or never.
Near-identical pages competing with one another.
robots.txt controls crawling, not indexing. Disallow a URL and Google stops fetching it — which also stops it reading the very tag that would have taken the page out. The URL can then stay in results, listed from its inbound links alone.
# robots.txt
User-agent: *
Disallow: /old/
The intent is to get the page out of Google.
GET /old/page
→ blocked by robots.txt
→ noindex never seen
The page is never fetched, so the noindex on it is never read.
example.com/old/page
No information is available for this page.
Indexed from inbound links, with no title or description because nothing was ever read.
Allow the URL, serve noindex, and wait for Google to recrawl it. Once the page has dropped out of the index you may disallow it, if you also want to save the crawl budget. The order matters: noindex first, disallow second, never both at once.
# robots.txt
User-agent: *
# no rule for /old/
# /old/page returns
<meta name="robots" content="noindex">
The same directive travels two ways. One is in the HTML, where anyone can see it. The other rides in the response headers, invisible in view-source, and is usually set by something nobody remembers configuring.
<head>
<meta name="robots" content="noindex, follow">
</head>
Shows up in view-source and in any browser inspector. Normally set deliberately, by a CMS toggle or an SEO plugin.
HTTP/1.1 200 OK
Content-Type: text/html
X-Robots-Tag: noindex
Nowhere in the page source. You only see it if you look at the response itself, which is why it survives redesigns and migrations unnoticed.
One request shows every header the server sends. If x-robots-tag comes back at all, that page is carrying a directive no amount of reading the HTML would reveal.
$ curl -sI https://example.com/page
HTTP/1.1 200 OK
x-robots-tag: noindex
This tool answers the first two. If both pass and the page still doesn't appear, the problem is relevance or quality, not access.
What the search engines document about each check this tool runs, with every claim linked to its source.
Both forms say the same thing [1][2]. They differ only in where the directive travels, which decides whether it works at all for a given file type.
noindex there. Then confirm none of them is disallowed in robots.txt, or the directive is never seen [1].<meta name="robots" content="noindex">Only read inside
<!-- or addressed to Google alone -->
<meta name="googlebot" content="noindex">
<head>. A tag written into the body by a script is ignored.HTTP/1.1 200 OKThe only option for PDFs, images and other non-HTML files, which have no
Content-Type: application/pdf
X-Robots-Tag: noindex
<head> to hold a tag [2].A disallowed URL can still appear in results if something links to it, shown with no description [3]. This is the most common way the two rules get confused.
noindex. Use Disallow only to save crawl budget on pages already excluded another way. Applying both cancels the first: a disallowed page is never fetched, so its noindex is never read [1].User-agent: *Intended to keep the whole section out of Google.
Disallow: /private/
example.com/private/pageIndexed from inbound links alone. Google never fetched it, so there is no title or description to display.
No information is available for this page.
4XX content is ignored, and URLs already indexed that begin returning 4XX are dropped over time [4].
GET /product → 301The destination is what counts. A chain ending at a 404 removes the page as surely as serving the 404 directly [4].
GET /produkt → 302
GET /p/123 → 404
GET /product → 301One hop to a page that returns 200. Keep redirects short; every extra hop is another chance to break [8].
GET /p/123 → 200 OK
rel="canonical" is a strong hint, not an order, and Google may pick a different URL [5].When this status appears the page is working correctly and is simply not the version chosen. That is fine when deliberate, and a silent loss when not [9].
<!-- on /blue-widget -->This page asks Google to index a different URL. Copied templates are the usual cause: every variant names one page, and only that page is indexed.
<link rel="canonical"
href="https://site.com/widget">
<!-- on /blue-widget -->Absolute URL, naming itself, matching the URL you want to appear in results [9].
<link rel="canonical"
href="https://site.com/blue-widget">
This is a crawl-capacity decision, not a verdict on the page. Google limits how much it fetches from a site based on how quickly the server responds and how much it judges the site is worth crawling [10].
URL in sitemap → yesNo tag will clear this. The page has never been read, so there is nothing on it to fix.
Fetched by Google → no
Blocked / noindex → no
Verdict → queued
• slow server responsesEach one spends the same crawl capacity on pages that do not deserve it [10].
• thousands of near-identical URLs
• filters generating endless paths
• a new site with few inbound links
This is the hardest status to act on, because it is the only one that is not a setting. Google documents that it indexes pages it considers useful and distinct, not every page it can reach [11].
noindex it deliberately rather than leaving it in limbo [11].Blocked by robots.txt → noEvery technical gate passed. Re-checking tags will not move it.
noindex present → no
Bad status code → no
Canonical elsewhere → no
• thin or templated pagesThe page is reachable but not distinct enough, or not important enough on your own site, to earn a slot.
• near-duplicates of a stronger page
• auto-generated tag and filter pages
• no internal links pointing to it
Google crawls the HTML, queues the page, then renders it and indexes the result [12]. A page can pass every check above and still be indexed nearly empty.
<div id="root"></div>No content in the initial HTML. Everything depends on the script running successfully.
<script src="/app.js"></script>
<div id="root"></div>A blocked resource, a script error or a request that times out leaves the page empty at render time [12].
<!-- nothing else -->
Google uses sitemaps to find URLs it might otherwise miss, especially on large or poorly linked sites [13]. Inclusion settles nothing on its own.
noindex or a redirect can never appear in it. Keep lastmod honest, and submit it in Search Console so the Page indexing report can be read per sitemap [7].<loc>https://site.com/old</loc>Every contradictory entry spends crawl capacity and weakens the signal the file is meant to carry.
<!-- /old returns 301 -->
<loc>https://site.com/tag/x</loc>
<!-- /tag/x carries noindex -->
• returns 200A sitemap listing only indexable URLs is a far stronger signal than one listing everything [13].
• self-referencing canonical
• no noindex
• a page you would link to yourself
A stray noindex, a robots.txt line or a canonical pointing the wrong way sits in a template, and the template renders thousands of URLs. Checking one page tells you the fault exists. The full audit runs these five gates plus 240+ checks across eight areas on every URL you have, then groups the failures by the setting that caused them.
Four thousand pages excluded by one noindex is a single finding with one fix, not four thousand rows to read through.
Search Console reports what Google already crawled, on a delay of days or weeks. The audit crawls on demand, so a bad deploy surfaces now rather than after the pages have dropped out.
Each finding names the gate it fails, the Google documentation behind it and the edit that clears it, so nothing sends you back to search for what the status meant.