robots.txt controls what search engines crawl, and a single over-broad rule can hide your whole site. Here's how it actually behaves, why blocking isn't the same as deindexing, and how not to block your own pages.
It's a crawl-management tool, not an indexing tool. That one distinction explains almost every robots.txt mistake.
robots.txt is a plain text file at yourdomain.com/robots.txt that crawlers read before crawling. With User-agent, Disallow and Allow rules you tell bots which paths to skip, and you declare your sitemap. A sensible file looks like this:
You'd typically block admin areas, internal search, login/cart pages, and infinite parameter URLs that would waste crawl budget. What robots.txt does not do is remove pages from the index, and conflating those two jobs is where most sites get into trouble.
The most important thing to understand about robots.txt.
If you Disallow a URL, crawlers won't fetch its content, but Google can still index the URL if it's linked from elsewhere, showing it in results without a description ("No information is available for this page"). So disallowing a page you wanted hidden can leave a bare, description-less listing in search.
Here's the trap that catches people: if you want a page out of search, you might add a noindex tagand Disallow it in robots.txt to be thorough. But because robots.txt blocks crawling, Google never fetches the page, so it never sees the noindex, and the page stays indexed. To deindex a page, you must allow crawling so Google can read the noindex. Disallow and noindex work against each other.
Real robots.txt checks the audit runs, and how each goes wrong.
Disallow: / from staging. Staging sites often block all crawlers. If that file ships to production, your entire site becomes uncrawlable. This is the robots.txt equivalent of the stray noindex. Blocking indexable pages. An over-broad path rule can cover pages you want ranked, the audit flags robots.txt blocking indexable pages. Blocking the sitemap. A rule that disallows your sitemap path stops crawlers reading it. No sitemap declared. robots.txt should include a Sitemap: line pointing to your XML sitemap. Blocking CSS/JS. Google renders pages; blocking the assets it needs to render can hurt how it sees the page. Missing robots.txt entirely isn't fatal, but a present, correct file is best practice, the audit flags a missing robots.txt too.
robots.txt lives at the root of the domain, at https://yourdomain.com/robots.txt, and nowhere else. A crawler will not look for it in a subdirectory. The rules it contains apply only to the host that served it, so every subdomain needs its own file: a robots.txt on the main domain says nothing about shop.yourdomain.com. The file must return a 200 status and be served as plain text. A robots.txt that 404s is treated as "crawl everything"; one that returns a 500 can cause Google to pause crawling the site altogether.
To keep a page out of search results, allow it to be crawled and add a noindex meta tag or an X-Robots-Tag header. This is the opposite of most people's instinct, but it is the only combination that works: Google has to fetch the page to see the noindex directive, and a robots.txt Disallow prevents exactly that fetch. Blocking the URL instead leaves it eligible to appear in results, stripped of its description.
For anything genuinely sensitive, use authentication rather than either mechanism. Disallowed paths are publicly readable in robots.txt, so listing a private directory there advertises its location to anyone who looks. robots.txt is a crawl-management tool, not a security control and not a removal guarantee.
Add a Sitemap: https://yourdomain.com/sitemap.xml line to the file. It is the simplest way to make sure every crawler, not just the ones you have verified a property with, can find your sitemap. The directive is independent of the user-agent groups above it, so its position in the file does not matter. The Kalenux audit flags a robots.txt that does not declare one.
Free to start. Check your robots.txt against every indexable page on your site.
Start my free audit