robots.txt and XML Sitemaps for a Business Website

What each file does

Google's introduction to robots.txt says the file "tells search engine crawlers which URLs the crawler can access on your site", mainly to avoid overloading the site with requests. It is about crawling: which addresses a crawler may fetch.

A sitemap works the other way round. Google's sitemaps overview describes it as a file with information about the pages, videos and other files on your site, which search engines read "to crawl your site more efficiently". It points crawlers towards the pages you consider important, and can say when each was last updated.

Neither file decides whether a page ranks. robots.txt can keep crawlers away; a sitemap can help them find pages. Whether a page is indexed and how it ranks depends on the page itself.

How a robots.txt file works

Google's guide to writing and submitting a robots.txt file sets out the basics:

• It is a plain text file named robots.txt, saved as UTF-8, and a site can have only one.
• It must sit at the root of the host it applies to, such as https://www.example.com/robots.txt, not in a subfolder. It covers only that protocol, host and port, so a subdomain needs its own file.
• It is made of groups. Each group starts with a User-agent line naming the crawler, followed by Disallow and Allow lines with paths that start with a slash.
• Paths are case-sensitive, and anything not disallowed may be crawled.
• A Sitemap line, with the full address of your sitemap, can go anywhere in the file.

A typical business site needs very little:

User-agent: *
Disallow: /cart/
Disallow: /search/

Sitemap: https://www.example.com/sitemap.xml

That lets every crawler in except for the cart and internal search results, and names the sitemap. If you have nothing to keep crawlers out of, a file with only the Sitemap line is fine. Google notes that the * group matches all crawlers except its AdsBot crawlers, which have to be named explicitly.

When two rules conflict, Google's page on how it interprets the robots.txt specification says crawlers use the most specific rule, based on the length of the path, and in a tie Google uses the least restrictive one. So Allow: /services/ wins over Disallow: / for your service pages.

If your site runs on a hosted platform, you may not be able to edit robots.txt at all. Google says such platforms may offer a search settings page instead.

What robots.txt cannot do

This is where most misunderstandings start. Google says robots.txt "is not a mechanism for keeping a web page out of Google". A page you disallow can still be indexed if other pages link to it; Google simply has not read it. Its address can then appear in results without a description.

It is not security either. The robots.txt standard, RFC 9309, published in September 2022, says plainly: "These rules are not a form of access authorization." Its security section adds: "Listing paths in the robots.txt file exposes them publicly and thus makes the paths discoverable." Anyone can open your robots.txt, so listing /admin-old/ or /private-files/ there advertises them. Respectable crawlers follow the rules; others may not. Private content needs a password or login.

Be careful with resource files too. Google says you can block unimportant images, scripts or style files, but not ones that make your pages harder for its crawler to understand when they are missing, or Google will not do a good job of analysing those pages.

Disallow or noindex?

To keep a page out of search results, Google's page on blocking indexing with noindex points to a noindex rule: either a meta tag in the page's head,

<meta name="robots" content="noindex">

or an X-Robots-Tag: noindex HTTP header, which also works for PDFs and other non-HTML files. When Google crawls the page and sees the rule, it drops the page from its results, even if other sites link to it.

The catch: the page must not be blocked in robots.txt. If crawlers cannot fetch it, they never see the noindex rule, and the page can stay in results. Putting noindex inside robots.txt does not work either: "Specifying the noindex rule in the robots.txt file is not supported by Google."

In short:

• Disallow in robots.txt for areas crawlers should not spend time on, such as cart pages, internal search results or endless filter combinations.
• noindex for pages people can reach but that should not appear in search, such as thank-you pages or thin tag archives. Leave them crawlable.
• A password or login for anything that must stay private, including staging sites.

robots.txt mistakes that hurt

• A leftover Disallow: / from development. Staging sites are often blocked this way, and the file gets copied to the live site. Our website launch checklist includes this check.
• The file returns a server error. Google's specification page says that if robots.txt returns a 5xx error, Google stops crawling the site for the first 12 hours while it keeps trying, then uses the last good copy for up to 30 days. A 4xx error other than 429 is treated as if there were no robots.txt, meaning no restrictions.
• Expecting changes to apply at once. Google generally caches robots.txt for up to 24 hours. If you need it refreshed sooner, Google documents how to submit an updated file.
• Rules Google ignores. Google supports user-agent, allow, disallow and sitemap; other fields such as crawl-delay are not supported. Google reads only the first 500 KiB of the file and ignores anything after that.
• Wrong case or a missing slash. Disallow: /Private/ does not block /private/.

Once the file is live, Google points to the robots.txt report in Search Console for finding and fixing problems. Our Google Search Console guide covers setting up the property.

Do you need an XML sitemap?

Not always. Google's overview says that if your pages are properly linked, Google can usually discover most of the site. It lists cases where you might not need one: a site of about 500 pages or fewer that matter for search, linked so every important page can be reached from the home page, without many videos, images or news pages you want in results.

A sitemap helps more when the site is large, new with few links from other sites, or rich in video and images. Even then, Google says a sitemap "doesn't guarantee that all the items in your sitemap will be crawled and indexed". Most content management systems, WordPress included, already produce one; check before you build your own.

What goes in a sitemap

The XML format is defined by the sitemaps.org protocol. Each page gets a <url> entry with a <loc> address; every other tag is optional. Google's guide to building a sitemap adds its own rules:

• Full, absolute addresses, such as https://www.example.com/services/, not /services/. Google crawls them exactly as listed.
• Only pages you want in search results, in their canonical form. Leave out redirected, noindexed and duplicate addresses.
• An honest lastmod. Google uses <lastmod> if it is consistently and verifiably accurate. It should reflect the last significant change to the page, such as the main content, structured data or links; a new copyright year does not count.
• No need for priority or changefreq. Google's words: "Google ignores <priority> and <changefreq> values."
• Size limits. One sitemap can hold up to 50,000 URLs or 50MB uncompressed. Larger sites split the list and use a sitemap index file that lists the individual sitemaps.
• One host per sitemap, UTF-8 encoded, placed at the root so it can cover the whole site.

A short example:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://www.example.com/services/</loc>
    <lastmod>2026-10-01</lastmod>
  </url>
</urlset>

Keep the sitemap in step with the site. When you redesign or move pages, the old addresses should come out and the new ones go in; our guide to keeping SEO through a redesign covers the rest of that work.

Telling search engines about the sitemap

Google lists three ways: the Sitemaps report in Search Console, the Search Console API, or a Sitemap line in robots.txt, which Google finds the next time it reads the file. Google says there is no limit to the number of Sitemap lines in robots.txt. Submitting in Search Console shows when Google read the file and any processing errors; our Search Console guide walks through it. For Bing, see our guide to Bing Webmaster Tools and IndexNow.

A quick check for your site

• Open yourdomain.com/robots.txt in a private browser window. It should load, with no Disallow: / under User-agent: *.
• Make sure nothing private is listed there as a way of hiding it.
• Check that the Sitemap line uses the full address and that the sitemap opens.
• Spot-check the sitemap: live, canonical pages only, no redirects or test pages, sensible lastmod dates.
• Confirm pages you want out of search use noindex and are not also disallowed.
• Look at the robots.txt and Sitemaps reports in Search Console for errors.

Frequently asked questions

Does every website need a robots.txt file?

No. Without one, crawlers may crawl the whole site. A file is useful when there are areas you want crawlers to skip, or to point them to your sitemap.

Will robots.txt keep a page out of Google?

Not reliably. Google says a disallowed page can still be indexed if other pages link to it. Use a noindex rule on a crawlable page, or put the page behind a password.

Should I list my admin or private folders in robots.txt?

No. RFC 9309 notes that listing paths in robots.txt makes them discoverable, and the rules are not a form of access control. Protect private areas with a login instead.

Do I need an XML sitemap for a small business site?

Often not. Google says a site of about 500 pages or fewer, well linked from the home page, may not need one. It also says that in most cases a site will benefit from having one, and many content management systems create it for you.

Do priority and changefreq help my rankings?

No. Google ignores both. It uses lastmod when the dates are consistently and verifiably accurate.

Want a site built with the search basics in place, from a clean robots.txt and XML sitemap to redirects from old addresses? See our website development service and all our services, or email [email protected].