Learn line

Add a sitemap to robots.txt

One line, Sitemap: followed by the full URL, lets every major search engine find your sitemap without you submitting it anywhere. Build the file below and see which parts matter.

Where to put the Sitemap line

Both work identically: Sitemap: is not tied to any user-agent group, wherever you write it. Putting it at the end just makes that obvious to humans.

User-agent: *
Disallow: /cart
Disallow: /search

Sitemap: https://example.com/sitemap-index.xml

The rules for the Sitemap line

  • Absolute URL only. Sitemap: /sitemap.xml is invalid; write Sitemap: https://example.com/sitemap.xml.
  • Not tied to a group. It applies to every crawler no matter where it appears. Many people write it last so nobody assumes otherwise.
  • As many as you like. List each sitemap on its own line, or list one index. Both are fine; the index is easier to maintain.
  • robots.txt must be reachable. It lives at /robots.txt on each host (and protocol) separately. blog.example.com needs its own.
  • Don't block what you list. A URL in your sitemap that robots.txt disallows is a contradiction. Search Console reports it as “Submitted URL blocked by robots.txt”.

Want to see how a live site does it? The sitemap finder reads any domain's robots.txt and reports its Sitemap lines.

Questions people ask

How do I add a sitemap to robots.txt?
Add a line Sitemap: https://example.com/sitemap.xml anywhere in the file. Use the full absolute URL. You can list several sitemaps, one per line, or list a single sitemap index.
Does the Sitemap line need to be under User-agent?
No. The Sitemap: directive is independent of user-agent groups; every crawler that supports it reads it, wherever it appears in the file.
Can robots.txt point to a sitemap on another domain?
Yes. Under the sitemaps.org cross-submission rules, a Sitemap: line in example.com’s robots.txt may point to a file hosted elsewhere (for example on a CDN or a sitemap subdomain). That tells crawlers you control the URLs listed.
Is robots.txt enough, or do I still submit in Search Console?
Google, Bing and others read the robots.txt line, so it works without logging in anywhere. Submitting in Search Console as well gets you the status and indexing report for each sitemap.
Can I block the sitemap file itself in robots.txt?
You can, but then crawlers are not allowed to fetch it. Never disallow the path of your own sitemap.

Next stops

Submit to GoogleSearch Console steps and what each status means.Sitemap finderFind where a site keeps its sitemaps.Sitemap generatorTurn a URL list or CSV into valid XML sitemaps.What is a sitemap?How crawlers use sitemaps, and whether you need one.

Sources