The meta robots tag is one of the most powerful and most misused tags in SEO. An accidental noindex can deindex important pages. A nofollow in the wrong place can prevent PageRank from flowing to pages that need it. This guide covers all available directives, their combinations, and the most common mistakes.
Meta robots tag syntax
<!-- Basic syntax --> <meta name="robots" content="directive1, directive2" /> <!-- To target a specific bot --> <meta name="googlebot" content="noindex" /> <meta name="bingbot" content="noindex" /> <!-- Common examples --> <meta name="robots" content="noindex, nofollow" /> <meta name="robots" content="index, follow" /> <!-- default behavior --> <meta name="robots" content="noindex, follow" /> <meta name="robots" content="noarchive" />
The main directives and what they do
| Directive | What it tells Google |
|---|---|
| index | Index this page (default behavior, no need to declare it) |
| noindex | Do not index this page, do not show it in search results |
| follow | Follow the links on this page and pass PageRank (default behavior) |
| nofollow | Do not follow the links on this page or pass PageRank |
| noarchive | Do not save or show a cached version of this page in results |
| nosnippet | Do not show a text snippet or video preview in results |
| noimageindex | Do not index the images on this page |
| none | Equivalent to noindex, nofollow simultaneously |
| all | Equivalent to index, follow (default behavior) |
When to use noindex
Noindex is the most used directive. It tells Google not to include that page in the index — the page remains crawlable but will not appear in search results.
- •User pages: cart, checkout, my account, order history.
- •Confirmation pages: thank you for your purchase, form submitted.
- •Admin or preview pages accidentally accessible.
- •Internal site search result pages.
- •Deep pagination pages without exclusive content.
- •WordPress tag pages with very few posts.
- •PPC landing pages you do not want to rank organically.
Noindex vs. nofollow: the key difference
Noindex and nofollow are independent directives that can be combined in any way. It is a common mistake to use them as if they were synonyms.
| Combination | Google indexes the page | Google follows links | When to use it |
|---|---|---|---|
| index, follow | ✅ Yes | ✅ Yes | Normal behavior for SEO pages |
| noindex, follow | ❌ No | ✅ Yes | Page that should not be indexed but should pass PageRank |
| index, nofollow | ✅ Yes | ❌ No | Indexable page whose links are paid or untrustworthy |
| noindex, nofollow | ❌ No | ❌ No | Private or admin pages |
noindex, follow is the most underused combination. Use it on pages you do not want indexed but that link to important pages — PageRank still flows through the links even though the page does not appear in Google. Example: a pagination page with noindex but with links to products you want to rank.
Meta robots vs. robots.txt: critical differences
Meta robots and robots.txt are two different mechanisms with different behaviors. Confusing them is a serious SEO mistake.
| Characteristic | Meta robots (noindex) | Robots.txt (Disallow) |
|---|---|---|
| Level of application | Individual page | URL or entire directory |
| Google crawls the page | ✅ Yes (to read the noindex) | ❌ No (blocks access) |
| Google can index the page | ❌ No (respects noindex) | ⚠️ Yes (cannot read noindex if blocked) |
| Passes PageRank from links | Depends on follow/nofollow | No (cannot follow links) |
The most dangerous error: blocking a URL in robots.txt AND adding noindex to it. If Google cannot crawl the page (robots.txt Disallow), it also cannot read the noindex. The result is unpredictable — Google may index the page anyway if there are external backlinks referencing it.
Noarchive, nosnippet, and other lesser-known directives
noarchive
Prevents Google from showing the "Cached" link next to the search result and from saving a cached copy. Useful for pages with sensitive information that updates frequently (prices, availability).
nosnippet
Prevents Google from showing text fragments or video previews in results. The result appears without a description. Rarely useful from an SEO perspective — it reduces CTR.
max-snippet, max-image-preview, max-video-preview
These directives allow controlling the maximum length of the text snippet and the size of image and video previews. For example: max-snippet:50 limits the snippet to 50 characters. Useful for publishers who want to control how much content Google shows without a subscription.
How to verify the meta robots tag on your pages
iRankly's Meta Tags Previewer shows all meta tags for each analyzed URL, including meta robots. Use it to verify that pages that should be indexable do not have an accidental noindex, and that private pages have the correct configuration.
Try the tool for free
Meta Tags PreviewerAnalyze your URLs with {tool} by iRankly. No sign-up, no credit card.