Why are some user agents still getting a meta robots tag with noindex from my site?
A friend pointed out to me that a website is not in Google because of the presence of a
meta name='robots' content='noindex,follow' /
tag in the home page. Ok, should be easy enough to turn that off, right?
Wrong.
Site uses Genesis framework, and in the SEO settings, none of the options are enabled to generate noindex.
Go to the individual page (homepage is a page, not a post), look there, none of the noindex options are checked there either.
Ok, one thing I do see, something is generating a robots.txt with disallow all. Not sure what is doing that, but it's not helping, so I upload a new robots.txt and remove write perms and now I can see my new robots.txt is remaining in effect that allows spiders.
Still getting that tag in the header though.
Then I think, has to be the coming soon plugin, (Soon) just being wonky, so disable that completely, still have the noindex tag.
(Then I remember I've never cleared the wp cache on any of these attempts, so do that....)
Then I find this Remove meta robots tag from wp_head
and realize wait, what? There's a blog_public wp_option?
Install phpMyAdmin so I can look, sure enough, blog_public = 0. EUREKA?
Set that to 1, reload in Firefox and Chrome, the noindex tag is gone (victory?)
Head over to Google webmaster tools, tell it to reindex and it's still complaining it won't because there's a robots tag with noindex
W-T-actual-F
In a shell, now...
lynx -source -dump https://mysite.example
html lang="en-US"
head itemscope itemtype="https://schema.org/WebSite"
meta charset="UTF-8" /
meta name="viewport" content="width=device-width, initial-scale=1" /
**meta name='robots' content='noindex,follow' /**
Where on earth it that coming from? Considering I can still reload the page in FF or Chrome and that tag doesn't show up?
Topic noindex robots.txt Wordpress
Category Web