Meta Robots

The meta robots tag is an HTML meta tag that’s found within the HEAD section (the top) of a webpages HTML code, it’s useful for advising search engines like Google how to index a webpage.

The robots meta tag can be used to prevent (but NOT encourage) search engine spiders from accessing individual webpages.

Robots Meta Tags

• <meta name=”robots” content=”index, follow”>
• <meta name=”robots” content=”noindex, follow”>
• <meta name=”robots” content=”index, nofollow”>
• <meta name=”robots” content=”noindex, nofollow”>

In the wrong hands the robots meta tag can cause a lot of damage to a websites rankings, the very popular WordPress SEO Plugins Yoast and All In One SEO (millions of users) both add meta robots tags to multiple sections of a WordPress blog (categories and tags for example) without explaining the potential SEO damage.

If a webpage lacks a robots meta tag the default is the webpage is indexed (Google indexes it) and followed (Google follows all links from the page), if your intention is for Google to have maximum access to a website (you want everything indexed and all links followed) there’s no reason to add any meta robots tags.

To reiterate, if your intention is for Google and other search engines to have full access to your website you don’t need ANY meta name robots tags.

Let’s start with the default.

<meta name=”robots” content=”index, follow”>

What you put inside the meta robots tag (if you use one) informs search engines what to do with the webpage regarding indexing the page and following it’s links.

If Google etc… doesn’t find a meta robots tag it will index the webpage and follow all it’s links. You will find many websites use the code below and it indicates the webmaster doesn’t understand basic SEO because it’s the default:

I’ve been working in SEO for over 15 years, I’ve built hundreds of websites with millions of webpages, all my income is basically from websites and I NEVER use the above robots meta tag and would NEVER advise using it. It is pointless adding an index, follow meta robots tag, you DO NOT need to add it just in case!

If you care about performance SEO which is part of the Google ranking algorythm you don’t add unnecessary code just in case!

I’ve added it to this SEO tutorial for clarity, but if you go ahead and use I’m embarrassed for you!

The default index, follow code is also equal to:

The case is ignored.

If you only include one option the default is used for the other option, so

is equal to the default index, follow.

is also equal to the default index, follow.

You will also find some websites list all as an option:

I don’t believe search engines recognize the all option, but it doesn’t matter since if a robots meta tag isn’t valid, it’s ignored and index, follow is used.

It’s no different to having a made up robots meta tag like this:

Google etc… will treat the above as equal to index, follow. :-)

I just wrote half an article telling you not to use the index, follow robots meta tag!

Let’s get to some useful robots meta tag options.

<meta name=”robots” content=”noindex, follow”>

The noindex, follow meta robots tag means:

Do NOT index this webpage.
Do follow the links to other webpages.

This meta robots tag is equal to:

The above is the best SEO option for stopping Google etc… from indexing a webpage where you want all the links from the webpage followed so the rest of the site is indexed/ranked.

This can be useful on webpages Google etc… can get access to, but shouldn’t be indexed. For example webpage related to a shopping cart basket where the page shows what a visitor has added to their shopping basket.

Note: Google honors the noindex robots meta tag, but this does NOT mean Google won’t list the webpage in Google. I know it’s confusing, but if a webpage with a robots noindex meta tag has a fair number of backlinks Google will index the page based on the backlinks. What you will find is a noindexed page can be found by some Google searchers and the results page shows a link to the noindexed page using the pages URL as the title and a blank description.

<meta name=”robots” content=”index, nofollow”>

The index, nofollow meta robots tag means:

Do index this webpage.
Do NOT follow the links to other webpages.

This meta robots tag is equal to:

You would use this robots meta tag on webpages you want Google to index, but do not want ANY of the links from the webpage (that includes all internal and all external links) to pass any link benefit (PageRank/PR).

Before using the nofollow robots meta tag be aware nofollow deletes PR/link benefit. This means any pages with this robots meta tag are a dead end for PageRank flow, your internal web pages linked from a page with the above won’t gain any SEO boost from the links!

In 15 years plus working in SEO I’ve never used a nofollow meta robots tag. The problem with nofollow is it deletes the link benefit (the PR) that would normally flow through the links! All your hard work gaining backlinks is wasted on a page with a nofollow robots meta tag!

There are few webpages on website where you might want to block Google from passing SEO benefit to ALL of them including ALL internal links: I can’t think of a single real world use.

If you must block SEO benefit from passing through a link use the rel=”nofollow” attribute on a link directly. The rel=”nofollow” attribute also deletes link benefit/PR, but at least it’s one link at a time not ALL links: see the rel=”nofollow” tutorial for details.

Quick SEO Usage Tip: had I used the nofollow robots meta tag on this page EVERY link from this webpage would be nofollowed. In comparison had I added a rel=”nofollow” attribute to the tutorial link above, ONLY that one link would be nofollowed.

<meta name=”robots” content=”noindex, nofollow”>

The noindex, nofollow meta robots tag means:

Do NOT index this webpage.
Do NOT follow the links to other webpages.

Use the noindex, nofollow meta robots tag on webpages you don’t want the search engines to have anything to do with, but remember pages with the above on are a COMPLETE waste of SEO link benefit, the page can not rank for anything and PR/link benefit flowing to the page is deleted!

Meta Name Robots Conclusion

In conclusion for most webpages there’s no good SEO reason to add any meta name robots tags, the above meta tags are ONLY required when you want webpages blocked (noindexed) from search engines or links from a webpage not followed (nofollowed) by search engines.

David Law

David Law : Technical SEO Expert with 20+ years Online Business, SEO, Search Engine Marketing and Social Media Marketing experience... Creator of multiple WordPress SEO Themes and SEO Plugins. Interests: wildlife, walking, environmental issues, politics, economics, journalism, consumer rights.

Website - SEO Gold Services