WordPress SEO Noindex Meta Tag article created March 2014 and is part of a series of tutorials related to the WordPress SEO Plugin by Yoast.
WordPress archives generate duplicate content that have the potential to cause SEO issues, some of which are easily mitigated at theme level (Stallion Responsive Theme has features built in to reduce duplicate content issues) and other issues that Google etc… normally manage with no extra input from website owners.
Yet there’s an increasing tendency in the WordPress SEO plugin community to advise the inappropriate use of Noindex, Follow Robots Meta Tags and the even worse PR deleting Noindex, Nofollow Robots Meta Tag.
WordPress archives include: Home page archives, categories, tags, author archives and dated archives (monthly archives for example).
What’s a Noindex, Follow Robots Meta Tag
meta name=”robots” content=”noindex, follow”
A noindex, follow robots meta tag is code to advise search engines like Google to not index this page, but do follow the links to other pages and pass link benefit (PageRank/PR). The code is loaded within the head section of a web page and looks like this when viewing the code source:
<meta name="robots" content="noindex,follow"/>
The above code can be added to WordPress archives using the popular Yoast and All In One SEO plugins and it’s very easy for beginners to WordPress SEO to use inappropriately.
Yoast SEO Plugin Noindex Paginated Archives
Why Noindex Web Pages?
meta name=”robots” content=”noindex, nofollow”
There are webpages you do not want indexed in search engines, for example with WordPress there’s no good reason to index /wp-login.php (that’s the WordPress login page) or anything under /wp-admin/ or /wp-includes/. WordPress by default adds the following robots meta tag to the /wp-login.php page and the lost password page /wp-login.php?action=lostpassword.
<meta name='robots' content='noindex,nofollow'/>
This is an appropriate use of noindex, though WordPress login robots meta tag includes a nofollow meta tag as well, the nofollow meta tag means no links on the webpage are followed and the PR/link benefit is completely lost (nofollow deletes PageRank!). This is unfortunate, a better solution would be:
<meta name='robots' content='noindex,follow'/>
And set the link to wordpress.org within the login pages content to rel=”nofollow” and the link to Lost your password /wp-login.php?action=lostpassword to rel=”nofollow” as well, but leave the link back to home as a dofollow link (don’t add a rel=”nofollow” attribute means dofollow). The above is better for WordPress users because it recovers 1/3rd of the PR/link benefit that’s flowing into the /wp-login.php pages.
WordPress Login Links Noindex, Nofollow Robots Meta Tag
Some WordPress themes do pass link benefit through the login links via sitewide widget links, adding the above robots meta tag etc… to the WordPress login webpage as described above would recover around 1/3rd of the link benefit that’s currently lost.
Stallion Responsive Theme users need not worry, the Stallion meta widget login links are not text links, they are get forms which means they don’t pass any link benefit. I’ve also added code to the meta widget so it only loads on the home page, so even if Google could follow the form links it’s not sitewide like you see with most WordPress themes.
Quick SEO Tip Nofollow Deletes Link Benefit
As you can see there are appropriate uses for noindex and even nofollow (nofollow deletes link benefit and I have dozens of articles and forums posts arguing not to use rel=”nofollow”): would be wrong for WordPress to add dofollow links to wordpress.org from the login pages as users can not disable them easily.
Unfortunately WordPress SEO plugins like Yoast WordPress SEO and All In One SEO have features to add robots noindex meta tags to WordPress archives!
WordPress SEO Noindex, Nofollow Robots Meta Tag
Why do SEO Experts Recommend Noindex, Follow Meta Tags for WordPress Archives?
The main reason for some SEO experts advising noindexing WordPress archives is potential duplicate content issues. Let’s examine duplicate content on archive sections of a WordPress blog.
There are themes like the WordPress default Twenty Fourteen theme which outputs entire articles on the archive parts of a site. What this means is your main post content is copied verbatim to every archive it’s part of, not only is this damaging for your visitors experience, but also potentially damaging to your main contents (your WordPress Posts) rankings.
WordPress by default breaks archive sections of a site into 10 post blocks (can be changed under “Settings” >> “Reading” : “Blog pages show at most”), when you install WordPress for the first time running Twenty Fourteen you will have home page archives and the most recent 10 posts are shown in full on the home page! The next 10 posts are shown on page 2 of the homepage archives and so on.
By adding entire copies of posts in sets of 10 to archives it’s generating a lot of duplicate content and not giving your users (especially mobile visitors) a good user experience: enter a website running Twenty Fourteen with the default WordPress settings on an iPhone and note how the content of the posts is hidden from view this is to prevent endless scrolling!
SEO problem is the content was still downloaded by your mobile device, so your mobile phone still had to download 10 full posts including all the images, embedded videos etc… and that’s Google SEO damaging (page loading speed is a Google ranking factor, large pages are at an SEO disadvantage) I’m currently running SEO tests on a default WordPress install running Twenty Fourteen and when viewing source of the home page it’s over 2,500 lines of code because of the full content on archives!
You may note I tend to write long articles, imagine 10 posts like this one with images loaded on the first page of a category!!!
WordPress Categories SEO Noindex
There are lots of WordPress themes, like Stallion Responsive which do not output entire copies of posts on the archive sections of the site, so it’s really easy to solve without resorting to noindex meta tags etc…
See any category of this website (Yoast WordPress SEO Tutorial for example) and you’ll see a short excerpt of the posts rather than entire posts. Although the post excerpts are still duplicate content, they are so short and Google is so good at determining ‘real duplicate content’ (content scraped from other websites) you are highly unlikely to be downgraded for internal duplicate content: I no longer even consider an excerpt of an internal page duplicate content when thinking about duplicate content SEO issues.
Google Search Console also reports some types of duplicate content on your website, are you seeing anything that suggests Google can’t cope with a small amount of duplicate content?
Unfortunately not all WordPress themes include this very basic article excerpt theme feature and those that show full articles in sets of 10 do run the risk of falling foul of Google duplicate content issues: not search engine ranking penalties, ranking mistakes: Google gets confused.
WordPress categories for example will tend to have more link benefit flowing through them (they tend to have more internal links) than an individual WordPress post: it’s a simple case of most WordPress sites have a sitewide category widget, most WordPress posts (and static pages) do NOT have a sitewide link.
This means when Google indexes a category with full content of articles it can sometimes rank the category above the main post. This is not a duplicate content penalty, it’s basic SEO 101, the category has more backlinks and PR flowing through it relative to the deeper content simply because of your average WordPress navigation link structure.
Removing the category from the equation by using a “noindex, follow” robots meta tag is not the appropriate SEO course of action. If your theme can’t show archive excepts on categories change themes (Stallion Responsive :-)). If you already use a theme that shows excerpts be aware noindexing the categories does not mean Google will now rank your deeper content where the categories used to rank, you’ve not removed the reason why the categories tend to gain more PR/link benefit and you shouldn’t even try to ‘fix’ this, it’s not broken.
WordPress SEO Tutorial Category SERP
Be grateful a page on your site has gained a SERP and take full advantage of it. For example right now (March 2014) I’m targeting a page at the SERP “WordPress SEO Tutorial” : not an easy SERP.
I’ve been targeting this WordPress SEO Article at the SERP, but the WordPress SEO Tutorial category is my best Google result (around 13th in Google).
WordPress SEO Tutorial Google SERP
Rather than noindexing the category and loosing that SERP position I’ve optimized the category further for the SERP: Stallion Responsive has advanced SEO features for categories. I’ll be very happy to see the WordPress SEO tutorial category on the first page of Google for that search result, had I used the Yoast WordPress SEO or other SEO plugin feature to noindex categories I’d be loosing Google traffic!
WordPress categories are awesome SEO wise, don’t noindex them and certainly don’t nofollow them!!! Same is true for WordPress Tags if you tend to use tags rather than categories, I have loads of free Google traffic going to categories.
David Law
Website - SEO Gold Services
Previous marketing company used some automated content plugin on my wordpress site. It created a lot of tags and pages that are very similar. Some of the tags are ranking right next to the post/page for the same keyword. Would it be safe to use Yoast to noindex all 2,652 tags? I’m worried about taking a huge ranking hit.
Thank you.
Short answer is no, it would not be safe to use Yoast to noindex all 2,652 tags, it would waste a LOT of PageRank/link benefit.
Long answer.
Noindex tells Google to not index the webpages with the noindex meta tags, but do waste the PageRank/link benefit the webpages would have used for ranking purposes if the noindex meta tag didn’t exist: that’s bad SEO wise.
What that means to you is, you’ll probably have tens of thousands of internal links to those tags and that’s costing a lot of PageRank/link benefit which is NOT recovered by adding a noindex meta tag!
The best SEO solution is make a list of all WordPress tags with traffic and pay attention to conserving the traffic to those WordPress tag pages.
You have two obvious choices.
1. Keep the WordPress tags with traffic as they are now and delete everything else.
This will protect the majority of the traffic to the tags with traffic and free up a LOT of link benefit which isn’t generating traffic directly (tags with no traffic have little value) which Google will within a couple of months use on the rest of your webpages: rest of the site might rank higher after deleting thousands of worthless tags.
You have to remember every webpage indexed on your site costs PageRank/link benefit to keep Google indexing them. When you delete webpages with no traffic it frees up the PageRank/link benefit which will now flow into more important parts of the website.
2. Delete all WordPress tags and for the tags with current traffic setup 301 redirects to the most relevant webpage.
This recovers all the wasted PageRank/link benefit and temporarily passes the SEO benefit to ranked tags to other parts of the site. It’s temporary since it’s highly unlikely any websites are linking to your tags, this assumes all the links to tags are internal links which will all be deleted if you delete the tags.
In your question you mentioned the ranked tags are ranking for content where you already have webpages targeting those SERPs. You could be experiencing keyword cannibalisation, this is where 2 or more webpages are competing for the same SERPs and Google is ranking the one with the worst content (WordPress tags) above the better content and weirdly this can mean you rank a little lower overall.
If you are finding WordPress tags are ranking above main content (Posts) then look to solution 2. By 301 redirecting the tags to the most relevant Posts (what’s been keyword cannibalised) you combine the SEO ranking value of the 2
A derivative of the above between solutions 1 and 2 is, follow solution 1 (delete all tags with no traffic keeping the tags with traffic) and 301 redirect the tags with traffic, but don’t delete the tags at first.
By keeping the tags with traffic AND 301 redirecting them to the most relevant content, this keeps the internal links to those tag pages. Google can still find the links to the tags, but redirects most of the SEO benefit to where they are redirected.
The benefit of this solution is it keeps the internal link benefit to the tags with traffic and as Google passes the SEO value to the content they are redirected to you can slowly delete the tags (usually a few months later): this should stop any potential short-term drop in traffic you can see after deleting content with traffic.
David Law
Hi David,
First of, great article you’ve got! Really appreciate the quality content. Its really a deep dive. May I ask where can we implement ? I’ve been reading it online but they point me to plugins. I want to implement it manually, to distinct individual HTML pages on my site. Do you know where can I find the head section for individual pages in wordpress?
Cheers
Chris
With WordPress there aren’t any individual WordPress HTML pages, so to add something like a robots meta tag to an individual WordPress Post or WordPress Static Page requires either:
A WordPress Plugin which specifically adds the robots meta tags to the Posts/Pages.
A WordPress Plugin which allows the user to add any code to the head section to specific Posts/Pages.
A WordPress Theme feature which specifically adds the robots meta tags to the Posts/Pages.
A WordPress Theme feature which allows the user to add any code to the head section to specific Posts/Pages.
Some custom code added to your current themes functions.php file which achieves the above (basically a “WordPress Theme feature which…”).
I use the Stallion Responsive WordPress SEO Theme (I developed it) on this site and one of the options is “A WordPress Theme feature which allows the user to add code to the head section to specific Posts/Pages”.
So if I wanted to do what you want to do I’d activate that feature and on the Posts Edit screen paste in a full robots meta tags code.
The code behind the feature is no different to installing a plugin, it’s just built into a theme. There’s probably code snippets around to achieve the same thing, but it would be simpler and better to use a well supported WordPress plugin which either only adds robots meta tags or adds custom code.
I currently have ZERO WordPress Posts and Pages with robots meta tags noindex or nofollow code. Noindex and nofollow robots meta tags are anti-SEO and should be avoided if possible.
If there’s a webpage you don’t want indexing/following first ask why does that webpage exist?
If it exists for a reason why do you want it noindexing/nofollowing?
For example I see websites noindexing About, Contact, Privacy Pages and that’s SEO stupid, why would any business want those pages not indexed!
If there’s parts of a site you don’t want indexing consider deleting them, for example the dated archives are really wasteful SEO wise (they cause SEO damage), the solution isn’t add a noindex tag, it’s delete the dated archives (don’t use the dated archives widget).
David Law