An XML sitemap lists the pages you want search engines to know about. A robots.txt file tells crawlers which addresses they may request. They are separate files with separate jobs.
An XML sitemap lists the pages you want search engines to find. A robots.txt file tells crawlers which addresses they may request. Neither makes a page appear in Google, and robots.txt is not a way to keep a page out of search results. Small, well-linked sites can often manage with the files their platform generates.
What an XML sitemap does
An XML sitemap is a file that lists the addresses of pages you want search engines to know about. Each entry holds the full address of a page, and it can hold the date the page last changed. The file sits on your own site, usually at an address such as yourdomain.com/sitemap.xml, and crawlers read it to learn what exists.
A sitemap is a hint, not an instruction. Google's documentation says that submitting one only suggests which addresses to look at. Google is not obliged to fetch the file, crawl every address in it or index every page. Google also finds pages by following links, so a sitemap supports good linking and does not replace it. The guide to internal linking covers the linking side.
Google reads XML sitemaps. It also accepts RSS or Atom feeds and plain text files with one address per line. The file must use UTF-8 encoding. A single file has limits on size and on the number of addresses, so large sites split the list across several files and name them in a sitemap index file. Check the current documentation for the limits.
Does your site need an XML sitemap?
Often it is optional. Google says that if your pages are properly linked, it can usually discover most of your site. It lists three cases where a sitemap helps most: a large site, a new site with few links from elsewhere, and a site with a lot of rich media or news content. It says you may not need one if your site is small, which it puts at roughly 500 pages or fewer, and your pages are well linked to each other. Check the current documentation, because Google may change the wording.
In practice a sitemap costs little. Many content management systems and store platforms make one for you, and Google's documentation says that if yours does, you may not need to do anything. A new site, or one with pages that few links reach, gains the most. A well-linked brochure site loses nothing by having one.
What goes in a sitemap and what stays out
List the pages you want people to find in search, and nothing else.
- Include live pages that return a normal success status and that you want in results.
- Use full addresses, including https and the exact host you use, such as the www form if that is your main one. Google says it tries to crawl addresses exactly as listed.
- List the preferred version of each page. If the same content can be reached at several addresses, put only the canonical one in the sitemap, and keep it consistent with the canonical tag on the page.
- Leave out pages that redirect, return a 404, carry a noindex tag or are blocked by robots.txt. The sitemap would be asking for a page that the site then refuses.
- Leave out thank-you pages, internal search results, test pages and addresses created by tracking parameters.
- Add a last-modified date only if you keep it accurate. Google uses the date when it is consistently and verifiably right, and says it ignores the priority and change-frequency fields.
Let your platform generate and update the file where it can. A file edited by hand goes out of date as soon as a page changes. If you do edit one by hand, set a reminder to review it whenever pages are added or removed.
What a robots.txt file does
A robots.txt file is a plain text file that tells crawlers which addresses on your site they may request. Google's documentation says its main use is managing crawler traffic, for example to avoid overloading a site with requests. It sits at the root of the site, at yourdomain.com/robots.txt, and a site has one such file per host.
The file is made of groups of rules. Each group names a crawler and lists the addresses it may or may not request. The table explains a short example.
| Line | What it tells crawlers |
|---|---|
| User-agent: * | The group of rules that follows applies to all crawlers, apart from a few special Google crawlers that must be named separately |
| Disallow: /checkout/ | Do not request addresses that start with /checkout/ |
| Allow: /checkout/help/ | An exception: addresses that start with /checkout/help/ may be requested, because the more specific rule wins |
| Sitemap: https://www.example.com/sitemap.xml | Where the sitemap is. This line needs the full address |
A few details matter. Paths start with a forward slash and are case-sensitive. A rule matches every address that begins with that path. Google supports the fields user-agent, allow, disallow and sitemap, and ignores others, such as crawl-delay. The file must be plain text in UTF-8, so do not write it in a word processor, which can add odd characters such as curly quotes.
The rules apply only to the protocol, host and port where the file sits. A file at https://example.com/robots.txt does not cover http://example.com/ or a subdomain such as shop.example.com. Each needs its own file if you want rules there. A site without a robots.txt file is simply open to crawling.
Robots.txt is not a way to keep a page out of Google
Blocking an address in robots.txt stops well-behaved crawlers from requesting it. It does not remove the address from search. Google's documentation says a disallowed address can still be indexed if other sites link to it, and it may then appear in results without a description. In the Page indexing report in Search Console, this shows as "Indexed, though blocked by robots.txt".
| What you want | What to use |
|---|---|
| Stop crawlers requesting a section of the site | A disallow rule in robots.txt |
| Keep a page out of search results | A noindex rule, as a meta tag or an HTTP header, on a page that crawlers can reach |
| Keep a page private | A login or password. The robots.txt file is public, and anyone can read it |
| Keep a test site out of search | A password on the test site |
One trap catches many sites. For a noindex rule to work, the crawler must be able to fetch the page and read the rule. If robots.txt blocks the page, the crawler never sees the noindex tag, and the page can still appear in results. Google also no longer supports a noindex rule written inside robots.txt itself.
Do you need both files?
Most small sites need an XML sitemap as a helper and a robots.txt file as a safeguard rather than a tool. The table shows typical cases.
| Site | XML sitemap | Robots.txt |
|---|---|---|
| Small brochure site, well linked | Optional but cheap, and usually generated by the platform | Often a short default file that blocks nothing important |
| New site with few links to it | Worth having, so crawlers learn what exists | A short default file is enough |
| Online store with filter and search pages | Worth having, listing the main product and category pages | Often used to keep crawlers away from endless filter and search addresses |
| Test or staging site | Not needed | Not a reliable lock, so use a password |
A short default file usually allows all crawlers and names the sitemap. Your platform may create it for you. Open it and read what it says instead of assuming.
How to submit an XML sitemap to Google
- Find your sitemap address. Try yourdomain.com/sitemap.xml, look for a Sitemap line in robots.txt, or check your platform's SEO settings. Some platforms use an index file with a different name.
- Open the address in a browser. You should see a list of page addresses, or a list of further sitemap files. Check that the pages listed are the ones you want found.
- Add your site to Search Console and verify ownership if you have not already. The comparison of Google Analytics, Tag Manager and Search Console explains what Search Console is and how verification works.
- Open the Sitemaps report, enter the address of the sitemap file and select Submit. You need owner permissions on the property to do this. Google fetches the file, and you cannot upload one.
- Read the status. Success means Google loaded and processed the file. Has errors means it loaded but some entries failed. Couldn't fetch means Google could not load it. Common causes are a wrong address, a robots.txt rule blocking the file, a login in front of it or a server problem. If a fetch fails, Google retries for a few days and then stops, so fix the cause and submit again.
- Optionally add a Sitemap line to robots.txt. Google reads it there too, although the Sitemaps report lists only sitemaps submitted through the report or the API.
An older method, which asked Google to fetch a sitemap through a web address, no longer works. Google announced its end in 2023. Submitting through Search Console or robots.txt is the supported route. Submitting is a request, and Google gives no date by which pages will be crawled or indexed. Other search engines have their own webmaster tools with a similar step, so check their current documentation.
Mistakes that block pages by accident
- A disallow rule with only a slash after it, which blocks the whole site. It is common when a test site goes live with its settings unchanged, which is why the website launch checklist includes a robots.txt check.
- A noindex tag left over from the build, or a platform setting that asks search engines to stay away. This is not a robots.txt rule, but the result is similar: pages drop out of results or never enter them.
- A rule that matches more than you meant. Rules match every address that begins with the path, so a disallow for /s also blocks /services/ and /shop/. Write the full path and test it.
- Blocking a page and also tagging it noindex. The crawler cannot read the tag.
- Blocking the files that pages need to display, such as style sheets, scripts or images. Google's documentation says blocking such files is acceptable only if pages loaded without them are not significantly affected. If their absence makes a page harder for Google to understand, do not block them.
- Putting rules on the wrong host. A file covers only the host, protocol and port that serve it, not a subdomain or the other protocol.
- Listing blocked, redirected or noindex pages in the sitemap. The two files then disagree, and the Page indexing report in Search Console can warn about submitted addresses that are blocked.
- Forgetting to re-check after a redesign, a migration or a change of platform, when these files are most often overwritten.
How to check both files
After any change, check both files in a browser, then in Search Console.
- Open yourdomain.com/robots.txt and yourdomain.com/sitemap.xml, and read them.
- In Search Console, the robots.txt report shows the files Google found, when it last fetched them and any warnings or errors. It is available for domain-level properties. Google may keep a stored copy of the file for up to about a day, and the report has a recrawl request for emergencies.
- Use the URL Inspection tool on an important page. It shows whether crawling is allowed and whether indexing is allowed.
- In the Page indexing report, look for "Blocked by robots.txt" and "Indexed, though blocked by robots.txt" on pages that should be open.
- Crawl the site and compare the crawler's list with the sitemap.
The basic SEO audit guide places these checks within a wider review. If important pages are missing from the index and the reasons given do not explain why, the signs your website needs a technical SEO audit help you decide whether to call a specialist. The search engine optimisation service page describes the SEO work Liftup does.
Questions and answers
What is an XML sitemap?
An XML sitemap is a file that lists the addresses of the pages you want search engines to know about, and it can include the date each page last changed. It is a hint to search engines, not an instruction. Google says submitting one does not oblige it to crawl or index every address.
What is a robots.txt file?
A robots.txt file is a plain text file at the root of a site that tells crawlers which addresses they may request. Google says its main use is managing crawler traffic. It is not a way to keep a page out of search results.
Does my site need both an XML sitemap and a robots.txt file?
Not always. A small, well-linked site may not need a sitemap, though many platforms generate one anyway and it costs little. Robots.txt is needed only if you want to keep crawlers away from some addresses. Open the default file and check what it says.
How do I submit a sitemap to Google?
Add and verify your site in Search Console, open the Sitemaps report, enter the sitemap's address and select Submit. You need owner permissions. You can also list the sitemap in robots.txt. Check the status afterwards and fix any error before submitting again.
What mistakes block pages by accident?
The common ones are a rule that blocks the whole site left over from the build, a noindex tag that was never removed, a rule that matches more addresses than intended, and blocking files that pages need to display. Check robots.txt after every redesign or move.
Can a page blocked by robots.txt still appear in Google?
Yes. Google says a blocked address can still be indexed if other sites link to it, usually without a description. To keep a page out of results, use a noindex rule on a page that crawlers can reach, or put it behind a login.