Sitemap and robots.txt Done Properly

25 May 2026

A one-line typo in robots.txt can deindex your entire site, and nothing will tell you it happened until traffic quietly falls off a cliff over the next few weeks. Check yours right now:

curl -s https://yoursite.com/robots.txt

The mistake that deindexes everything

User-agent: *
Disallow: /

That’s it. That’s the whole site, gone from search, and it’s usually left over from a staging config that got copied into production and never removed. If you see a bare Disallow: / under User-agent: * in production, fix it immediately — every crawler respects this, including Google.

A correct, minimal robots.txt

User-agent: *
Disallow: /admin/
Disallow: /drafts/

Sitemap: https://yoursite.com/sitemap.xml

Disallow only what genuinely shouldn’t be crawled — admin routes, draft content, internal search result pages that create infinite duplicate URLs. Don’t disallow CSS or JS files; Google needs to fetch those to render the page and judge mobile usability. That used to be common advice and it actively hurts you now.

Disallow vs. noindex — they do different things

This is the mistake that confuses people most:

	`robots.txt Disallow`	`<meta name="robots" content="noindex">`
What it does	Tells crawlers not to fetch the page at all	Tells crawlers to fetch, but not index it
Page can still appear in search?	Yes — if linked from elsewhere, Google can index the URL with no content, showing just the URL	No — this is the actual way to keep a page out of search
Use for	Crawl budget management, keeping bots off truly irrelevant paths	Thank-you pages, internal search results, duplicate content you still want visitors to reach

If your goal is “don’t show this page in search results,” use noindex, not Disallow. A disallowed page with backlinks can still show up as a bare URL with no snippet — worse than not being indexed at all, because it looks broken.

The sitemap reference goes in robots.txt too

Most people submit the sitemap manually in Search Console and stop there. Also reference it in robots.txt — it’s how every other crawler (Bing, DuckDuckGo, AI crawlers that respect the standard) finds it without you submitting to each one separately:

Sitemap: https://yoursite.com/sitemap.xml

Verify the sitemap isn’t lying to you

curl -s https://yoursite.com/sitemap.xml | grep -c '<loc>'

Compare that count to your actual post count. If it’s wildly off, check for:

Draft posts (draft = true in front matter) that leaked into the build
A noindex param blocking sitemap generation for a whole section
Pagination or taxonomy pages inflating the count with low-value URLs

Staging leaks are the silent killer

If your staging environment is a subdomain with no auth and no robots.txt blocking it, Google will index it, and now you have duplicate content competing with your real site — sometimes outranking it if staging got crawled first. Password-protect staging, or at minimum ship a hard Disallow: / there specifically, never on production.

Check both files today. It takes thirty seconds and it’s the difference between “indexed” and “invisible.”

IAnswerable