How to prevent Google from indexing the /wp-content/ directory?

I received an error from google saying that I have mobile usability errors on my site.

the page that they are referring to is www.example.com/wp-content/plugins/ag-admin.

I'm not asking for help with that plugin, but trying to figure out how to make google not index that directory at all.

Is disallawing this in my robots.txt enough or should I even have to do that?

Topic robots.txt Wordpress

Category Web


Using robots.txt

Yes you could use a robots.txt file for this, simply add the following into this file:

User-agent: *
Disallow: /wp-content/

Notice that you can have multiple Disallow directives if you would like to restrict indexing of other folders as well.

User-agent: *
Disallow: /wp-content/
Disallow: /wp-admin/

If you would like to allow indexing on a specific file in the folder you could use an Allow directive after Disallow like this:

Disallow: /wp-content/
Allow: /wp-content/plugins/askimet.php

If you only would like to stop google from indexing you could instead add a user agent:

User-agent: Googlebot

Using set Header

You should also be able to use .htaccess for this. Try adding something like this in your .htaccess file:

<Directory wp-content>
  Header set X-Robots-Tag "noindex"
</Directory>

For the above to work you need to have mod_headers module enabled in apache. You can enable this module by executing:

sudo a2enmod headers
sudo apachectl restart

Here's the thing

There are multiple ways to do what you want, from adding meta tags, to passing headers, but because, you tagged your question with robots.txt So i consider it off-topic to discuss any other solutions.

Considering your demand you need to have this as your robots.txt This disallowed access to wp-admin, but depending on use case, you may need ajax, so I gave it an exception, and then disallowed wp-content, and wp-includes.

Change your robots.txt situated in root directory, like this

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Noindex: /wp-content/
Noindex: /wp-includes/

This shall ask google to not index anything in wp-content and wp-includes, Google is slow and it would take its bot some time, to realize he is going at wrong place, so it would eventually remove it from its index.

The below HTACCESS METHOD is completely optional, the Robots.txt can do the trick with work, but HTACCESS is much more consistent method, therefore I couldnt resist myself from telling it

Incase you are willing to edit ht-access file, then that would be a better approach.

I myself use it on my site. My code below if put in htaccess files, blocks all PHP and backend specific files, but allows all images, videos, pdfs and various similar file formats to be indexed by Google and others.

# Serves only static files
RewriteCond %{REQUEST_FILENAME} -f
RewriteRule ^wp-(content|includes)/([^/]+/)*([^/.]+\.)+ (jp(e?g|2)?|png|gif|bmp|ico|css|js|swf|xml|xsl|html?|mp(eg[34])|avi|wav|og[gv]|xlsx?|docx?|pptx?|gz|zip|rar|pdf|xps|7z|[ot]tf|eot|woff2?|svg|od[tsp]|flv|mov)$ - [L]
RewriteRule ^wp-(content|includes|admin/includes)/ - [R=404,L]

About

Geeks Mental is a community that publishes articles and tutorials about Web, Android, Data Science, new techniques and Linux security.