WordPress SEO Noindex Meta Tag article created March 2014 and is part of a series of tutorials related to the WordPress SEO Plugin by Yoast.
WordPress archives generate duplicate content that have the potential to cause SEO issues, some of which are easily mitigated at theme level (Stallion Responsive Theme has features built in to reduce duplicate content issues) and other issues that Google etc… normally manage with no extra input from website owners.
Yet there’s an increasing tendency in the WordPress SEO plugin community to advise the inappropriate use of Noindex, Follow Robots Meta Tags and the even worse PR deleting Noindex, Nofollow Robots Meta Tag.
WordPress archives include: Home page archives, categories, tags, author archives and dated archives (monthly archives for example).
What’s a Noindex, Follow Robots Meta Tag
A noindex, follow robots meta tag is code to advise search engines like Google to not index this page, but do follow the links to other pages and pass link benefit (PageRank/PR). The code is loaded within the head section of a web page and looks like this when viewing the code source:
<meta name="robots" content="noindex,follow"/>
Why Noindex Web Pages?
There are webpages you do not want indexed in search engines, for example with WordPress there’s no good reason to index /wp-login.php (that’s the WordPress login page) or anything under /wp-admin/ or /wp-includes/. WordPress by default adds the following robots meta tag to the /wp-login.php page and the lost password page /wp-login.php?action=lostpassword.
<meta name='robots' content='noindex,nofollow'/>
This is an appropriate use of noindex, though WordPress login robots meta tag includes a nofollow meta tag as well, the nofollow meta tag means no links on the webpage are followed and the PR/link benefit is completely lost (nofollow deletes PageRank!). This is unfortunate, a better solution would be:
<meta name='robots' content='noindex,follow'/>
And set the link to wordpress.org within the login pages content to rel=”nofollow” and the link to Lost your password /wp-login.php?action=lostpassword to rel=”nofollow” as well, but leave the link back to home as a dofollow link (don’t add a rel=”nofollow” attribute means dofollow). The above is better for WordPress users because it recovers 1/3rd of the PR/link benefit that’s flowing into the /wp-login.php pages.
Some WordPress themes do pass link benefit through the login links via sitewide widget links, adding the above robots meta tag etc… to the WordPress login webpage as described above would recover around 1/3rd of the link benefit that’s currently lost.
Stallion Responsive Theme users need not worry, the Stallion meta widget login links are not text links, they are get forms which means they don’t pass any link benefit. I’ve also added code to the meta widget so it only loads on the home page, so even if Google could follow the form links it’s not sitewide like you see with most WordPress themes.
As you can see there are appropriate uses for noindex and even nofollow (nofollow deletes link benefit and I have dozens of articles and forums posts arguing not to use rel=”nofollow”): would be wrong for WordPress to add dofollow links to wordpress.org from the login pages as users can not disable them easily.
Unfortunately WordPress SEO plugins like Yoast WordPress SEO and All In One SEO have features to add robots noindex meta tags to WordPress archives!
Why do SEO Experts Recommend Noindex, Follow Meta Tags for WordPress Archives?
There are themes like the WordPress default Twenty Fourteen theme which outputs entire articles on the archive parts of a site. What this means is your main post content is copied verbatim to every archive it’s part of, not only is this damaging for your visitors experience, but also potentially damaging to your main contents (your WordPress Posts) rankings.
WordPress by default breaks archive sections of a site into 10 post blocks (can be changed under “Settings” >> “Reading” : “Blog pages show at most”), when you install WordPress for the first time running Twenty Fourteen you will have home page archives and the most recent 10 posts are shown in full on the home page! The next 10 posts are shown on page 2 of the homepage archives and so on.
By adding entire copies of posts in sets of 10 to archives it’s generating a lot of duplicate content and not giving your users (especially mobile visitors) a good user experience: enter a website running Twenty Fourteen with the default WordPress settings on an iPhone and note how the content of the posts is hidden from view this is to prevent endless scrolling!
SEO problem is the content was still downloaded by your mobile device, so your mobile phone still had to download 10 full posts including all the images, embedded videos etc… and that’s Google SEO damaging (page loading speed is a Google ranking factor, large pages are at an SEO disadvantage) I’m currently running SEO tests on a default WordPress install running Twenty Fourteen and when viewing source of the home page it’s over 2,500 lines of code because of the full content on archives!
You may note I tend to write long articles, imagine 10 posts like this one with images loaded on the first page of a category!!!
WordPress Categories SEO Noindex
There are lots of WordPress themes, like Stallion Responsive which do not output entire copies of posts on the archive sections of the site, so it’s really easy to solve without resorting to noindex meta tags etc…
See any category of this website (Yoast WordPress SEO Tutorial for example) and you’ll see a short excerpt of the posts rather than entire posts. Although the post excerpts are still duplicate content, they are so short and Google is so good at determining ‘real duplicate content’ (content scraped from other websites) you are highly unlikely to be downgraded for internal duplicate content: I no longer even consider an excerpt of an internal page duplicate content when thinking about duplicate content SEO issues.
Google Search Console also reports some types of duplicate content on your website, are you seeing anything that suggests Google can’t cope with a small amount of duplicate content?
Unfortunately not all WordPress themes include this very basic article excerpt theme feature and those that show full articles in sets of 10 do run the risk of falling foul of Google duplicate content issues: not search engine ranking penalties, ranking mistakes: Google gets confused.
WordPress categories for example will tend to have more link benefit flowing through them (they tend to have more internal links) than an individual WordPress post: it’s a simple case of most WordPress sites have a sitewide category widget, most WordPress posts (and static pages) do NOT have a sitewide link.
This means when Google indexes a category with full content of articles it can sometimes rank the category above the main post. This is not a duplicate content penalty, it’s basic SEO 101, the category has more backlinks and PR flowing through it relative to the deeper content simply because of your average WordPress navigation link structure.
Removing the category from the equation by using a “noindex, follow” robots meta tag is not the appropriate SEO course of action. If your theme can’t show archive excepts on categories change themes (Stallion Responsive :-)). If you already use a theme that shows excerpts be aware noindexing the categories does not mean Google will now rank your deeper content where the categories used to rank, you’ve not removed the reason why the categories tend to gain more PR/link benefit and you shouldn’t even try to ‘fix’ this, it’s not broken.
WordPress SEO Tutorial Category SERP
Be grateful a page on your site has gained a SERP and take full advantage of it. For example right now (March 2014) I’m targeting a page at the SERP “WordPress SEO Tutorial” : not an easy SERP.
Rather than noindexing the category and loosing that SERP position I’ve optimized the category further for the SERP: Stallion Responsive has advanced SEO features for categories. I’ll be very happy to see the WordPress SEO tutorial category on the first page of Google for that search result, had I used the Yoast WordPress SEO or other SEO plugin feature to noindex categories I’d be loosing Google traffic!
WordPress categories are awesome SEO wise, don’t noindex them and certainly don’t nofollow them!!! Same is true for WordPress Tags if you tend to use tags rather than categories, I have loads of free Google traffic going to categories.