What Is the Meta Robots Tag?
The meta robots tag is an HTML directive that determines how a search engine bot indexes a single page and how it follows the links on that page. The <meta name="robots" content="..."> line added inside <head> gives Googlebot and other search engine bots page-specific instructions. For files that contain no HTML — images, PDFs and videos — the same job is done by the X-Robots-Tag HTTP header sent by the server.
This tag is typically used on pages you do not want to appear in search results: thank-you pages, duplicate URLs generated by filter and sort parameters, admin panels that require a login, or draft content not yet ready to publish. Used correctly, it helps the search engine direct its resources at the pages of your site that really matter.
Which Pages Should Use noindex?
Not every page needs to appear in search results; some pages are necessary for the visitor but gain nothing from being in the search index, and may even harm the site. Using noindex on pages like these helps the search engine reserve its resources for the pages of your site that genuinely create value.
- Pages reached only after an action, such as order confirmation or thank-you pages.
- Parameterised result pages produced by the internal search engine, such as
?q=URLs. - Product listing URLs spawned by filter and sort combinations that hold almost identical content.
- Account, cart and admin panel pages that require a login.
- Draft content not yet ready to publish, and duplicate pages in test/staging environments.
The Difference Between robots.txt and Meta Robots
robots.txt and the meta robots tag operate on two completely different layers. A Disallow rule in the robots.txt file prevents the bot from crawling the page; the bot never visits that URL at all. A meta robots tag containing noindex, by contrast, only works if the page has first been crawled: the bot visits the page, sees the noindex directive in the HTML, and only then decides to drop the page from the index.
That is why disallowing a page in robots.txt and adding noindex to it as well is a common mistake that defeats itself. Because the bot never crawls the page, it never sees the noindex tag inside it; if the page is linked from somewhere else, it can stay in the index regardless.
Meta Robots Directives: What Each One Does
Several comma-separated directives can be added to the content attribute. The most commonly used are:
| Directive | What it does |
|---|---|
| index / noindex | Determines whether the page appears in search results. |
| follow / nofollow | Determines whether <strong>all</strong> links on the page are followed. |
| noarchive | Prevents Google from showing a cached/copy version of the page. |
| nosnippet | Prevents a text or video preview from being shown in the search result. |
| noimageindex | Prevents the images on the page from being indexed in Google Images. |
| notranslate | Prevents Google from offering a translation of the page. |
| max-snippet:N | Sets the maximum character length of the preview text; <code>-1</code> means unlimited. |
Directives can be combined. To drop a page from the index entirely and stop its links being followed, for example, you would write noindex, nofollow; to index the page but turn off the cached copy and image indexing, index, follow, noarchive, noimageindex.
The HTML Tag and the X-Robots-Tag HTTP Header
The standard usage is to add a line such as <meta name="robots" content="noindex, nofollow"> to the <head> section of the page. That line applies only to that page and does not affect the rest of the site.
For non-HTML files such as images, PDFs and videos, a <meta> tag cannot be used; in that case the same directives are sent server-side with the X-Robots-Tag HTTP header. Google Search Central officially supports this method and accepts the same directive syntax (index/noindex, follow/nofollow, noarchive, max-snippet and so on). To drop every PDF file from the index on an Nginx server, for example:
The same rule is applied on Apache inside .htaccess or a virtual host definition like this:
nofollow: Page Level vs Link Level
The nofollow directive in the meta robots tag is a page-level setting and affects every link on the page. That is different from the rel="nofollow" attribute added to an individual link; that attribute affects only the single <a> element it is written on, not the other links on the page. Confusing the two can lead you to block the link value of an entire page unintentionally, or conversely to affect the whole page when you only needed to restrict a single link.
If you want to mark only a sponsored link in a blog post, for example, <a href="https://example.com" rel="nofollow"> is enough; the internal and external links in the rest of the page continue to be followed normally. A far broader decision such as stopping every link on the page from being followed can only be achieved with the page-level meta robots nofollow directive.
Common Mistakes
- Disallowing a page in
robots.txtwhile also addingnoindexto it — because the bot never crawls the page, the tag is never seen. - A
noindextag left behind in a test or development environment being carried into production with the code, silently dropping the page from the index. - Trying to add a
<meta>tag to non-HTML files such as PDFs — for those file types only theX-Robots-Tagheader works. - Confusing the page-level
nofollowdirective with therel="nofollow"attribute on an individual link.
Build the Tag With Our Tool
Instead of writing the directives by hand and risking a syntax error, you can pick the combination you want in the tool below and get both the HTML meta tag and the equivalent X-Robots-Tag HTTP header ready to use.