The noindex tag: meta robots and X-Robots-Tag

This guide is built from Google Search Central: Block Search indexing with noindex, the robots meta tag, data-nosnippet and X-Robots-Tag specifications, Remove a page hosted on your site from Google and JavaScript SEO basics, plus the Search Console Help article on the Page indexing report. Quotes are Google's.

Why a robots.txt block and noindex don't mix, and what belongs in a sitemap, are in our robots.txt and XML sitemaps guide. Hiding a URL for a few months while you fix it is in our Removals tool guide. This page is about the noindex rule itself: how to write it, where to put it, and how to check that Google has seen it.

What noindex does

noindex is a rule, set in a <meta> tag or an HTTP response header, that tells search engines which support it not to index a page. Google's wording is direct: when Googlebot "crawls that page and extracts the tag or header, Google will drop that page entirely from Google Search results, regardless of whether other sites link to it."

Three limits follow from how it works:

• Google has to crawl the page to see it. If robots.txt blocks the URL or the crawler can't reach it, "the crawler will never see the noindex rule, and the page can still appear in search results".
• It is a search rule, not access control. Google's removal page says a noindex tag "only blocks your page from showing up in Google search results. Users and other search engines that don't support noindex can still access your page."
• Other search engines may differ. Google notes that "some search engines might interpret the noindex rule differently", so a page could still show elsewhere.

Google also says noindex is useful "if you don't have root access to your server", because it works page by page.

The robots meta tag

For an HTML page, put this in the <head>:

<meta name="robots" content="noindex">

The name value robots means every crawler. To address only Google, use googlebot instead. Google supports two user agent tokens in this tag, googlebot (all text results) and googlebot-news (news results); "other values are ignored". Both the name and content attributes are case-insensitive.

Two details that catch people out:

• The robots rule "applies to search engine crawlers". Non-search crawlers such as AdsBot-Google need their own tag, for example <meta name="AdsBot-Google" content="noindex">.
• Google "doesn't enforce placement of meta robots in the HTML head" and also respects the tag in the body. The head is still the place to put it.

On WordPress, Wix or Blogger you may not edit the HTML directly. Google points to the CMS's search engine settings or a similar option. In WordPress, an SEO plugin usually adds a per-page setting for this.

The X-Robots-Tag header

The same rule can be sent as an HTTP response header. Google says the two methods "have the same effect", and "any rule that can be used in a robots meta tag can also be specified as an X-Robots-Tag". The header is the only option for files that have no HTML head: Google lists "PDFs, video files, and image files".

HTTP/1.1 200 OK
X-Robots-Tag: noindex

You set it in the web server configuration. Google's own example adds noindex, nofollow to every PDF on a site:

# Apache (.htaccess or httpd.conf)
<Files ~ "\.pdf$">
  Header set X-Robots-Tag "noindex, nofollow"
</Files>

# NGINX
location ~* \.pdf$ {
  add_header X-Robots-Tag "noindex, nofollow";
}

The header can name a crawler before the rules, for example X-Robots-Tag: googlebot: nofollow. Rules without a user agent "are valid for all crawlers". Header name, user agent and values are not case-sensitive.

	Robots meta tag	X-Robots-Tag header
Where it goes	In the page's HTML	In the HTTP response, set by the server
Works for	HTML pages	Any file: HTML, PDF, images, video
Who can set it	Anyone who can edit the page or CMS settings	Someone with access to the server or hosting config
Many URLs at once	Through a template or plugin	With a pattern match in the server config

Combining rules, and other rules worth knowing

Rules can go in one comma-separated list or in separate tags; both of these mean noindex plus nofollow:

<meta name="robots" content="noindex, nofollow">

<meta name="robots" content="noindex">
<meta name="robots" content="nofollow">

When rules conflict, "the more restrictive rule applies". When several crawlers are addressed, Google uses "the sum of the negative rules": a page with robots set to nofollow and googlebot set to noindex is read by Googlebot as noindex, nofollow.

Other rules from the same specification that sit close to noindex:

• none: "Equivalent to noindex, nofollow."
• nofollow: "Do not follow the links on this page." Without it, Google may use the page's links to discover other pages.
• unavailable_after: [date/time]: "Do not show this page in search results after the specified date/time." Useful for an event or offer page with an end date. Google adds that it "will decrease the crawl rate of the URL considerably" after that date.
• noimageindex: "Do not index images on this page."
• indexifembedded: lets Google index the page's content when it is embedded in another page through an iframe, "in spite of a noindex rule". It "only has an effect if it's accompanied by noindex".

Rules that limit snippets rather than indexing, such as nosnippet and max-snippet, are covered in our meta descriptions guide. Google lists noarchive and nocache as rules Google Search no longer uses.

One thing that doesn't work: a noindex line in robots.txt. "Specifying the noindex rule in the robots.txt file is not supported by Google."

noindex and JavaScript

You can add or change the robots meta tag with JavaScript, and Google gives an example of doing that on error pages in a single-page app. The other direction is unsafe. Google's JavaScript guide says: "When Google encounters the noindex tag, it may skip rendering and JavaScript execution, which means using JavaScript to change or remove the robots meta tag from noindex may not work as expected. If you do want the page indexed, don't use a noindex tag in the original page code." More on how Google renders scripts is in our JavaScript SEO guide.

How long it takes to drop out

A page leaves Google's results only after Googlebot recrawls it and sees the rule. Google's noindex page says: "If a page is still appearing in results, it's probably because we haven't crawled the page since you added the noindex rule. Depending on the importance of the page on the internet, it may take months for Googlebot to revisit a page."

You can ask for a recrawl with the URL Inspection tool. If the page has to go quickly, the Removals tool hides it from results "within a day", for about six months, while the noindex does the permanent work.

Checking that Google sees it

1. Look at what you are sending. Google suggests you "search the page source or response headers for the word" noindex. For a header, check the response in your browser's developer tools or with a command-line request.
2. Use URL Inspection. Google says to use it "to see the HTML that Googlebot received while crawling the page". Under Indexing allowed?, the report shows whether noindex is blocking indexing; Test live URL checks the current version. Our URL Inspection guide walks through the screens.
3. Watch the Page indexing report. Pages Google left out because of the rule are listed with the reason "URL marked ‘noindex’". Google's note on that status: "If you do not want this page indexed, congratulations! If you do want this page to be indexed, you should remove the 'noindex' directive." Our Page indexing report guide covers the other statuses.

If a page you want indexed shows up there, remove the rule, confirm with Test live URL, then use Request Indexing.

noindex, canonical, password or removal?

Goal	What Google points to
Page stays reachable but out of Google	noindex on a crawlable page
Two URLs show the same content	A canonical signal, not noindex. Google doesn't recommend noindex for choosing a canonical within a site, "because it will completely block the page from Search." See our canonical tags guide.
Only certain people should see it	Password protection, which keeps "Googlebot and other web crawlers" out while the right users can still view it
Page no longer exists	Remove it and return 404 or 410; see our 404s and redirects guide
Out of results today	Removals tool for the quick hide, plus one of the above to make it permanent
Don't crawl it at all	robots.txt, knowing that a blocked URL "can still appear in search results" without a description

Pages we usually noindex

This part is our own practice, not Google guidance. On business sites we build, we typically noindex: form thank-you and confirmation pages, internal site search results, login and account pages, thin tag or author archives a CMS creates on its own, and PDFs that duplicate an HTML page. We don't noindex product variants or filtered pages that are duplicates; those get a canonical. And the main check at every launch is the reverse problem: a staging site's site-wide noindex that gets copied to the live site. It's on our launch checklist for that reason, and it is the first thing we look at when a new site gets no search traffic.

When we set a noindex, we check it the same way each time: page source or headers, URL Inspection live test, then the Page indexing report a few weeks later.

Frequently asked questions

Should I block a noindex page in robots.txt as well?

No. Google has to crawl the page to read the rule. If robots.txt blocks the URL, Google never sees the noindex and the page can still appear in results when other pages link to it.

Is the meta tag or the X-Robots-Tag header better?

Google says they have the same effect. Use the meta tag for HTML pages you can edit, and the header for PDFs, images and other files, or for rules you want to apply to many URLs from the server config.

Why is my noindexed page still in Google?

Usually Google has not recrawled it since the rule was added, which can take months for less important pages. Request a recrawl with URL Inspection, check robots.txt is not blocking it, and use the Removals tool if it needs to go quickly.

Can I use noindex instead of a canonical tag for duplicate pages?

Google doesn't recommend it. noindex removes the page from Search completely, while a canonical tells Google which URL is the main one so the signals can be combined.

Does noindex stop links on the page from being followed?

No, noindex on its own only affects indexing. To ask Google not to follow the page's links too, add nofollow, or use none, which Google treats as noindex, nofollow.

Need indexing rules set up properly across a site, from page templates to server headers? See our website development and website maintenance services, all our services, or the rest of our guides.