Robots.txt not updating

Question

Robots.txt not updating

Confused

2020年6月26日 02:03

When I try to google search my web address, the follow comes up as the meta description

"A description for this result is not available because of this site's robots.txt – learn more."

The website I have tried to edit the robots.txt (downloaded the WP Robot plugin) and although I have changed it to

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/

The robots.txt still appears as

User-agent: *
Disallow: /

I have Yoast SEO, WP Robots, and Salient's Theme Nectar. If anyone might be able to provide me with a solution that would update the robots (or where to go from here) that would be incredibly helpful!

Topic robots.txt seo Wordpress

Category Web

Vahid · Accepted Answer · 2020年6月1日 10:16

1

Vahid answered at 2020年6月1日 10:16

I had the same issue and for me, I had to purge the Cloudflare cache.

Gonen · Accepted Answer · 2019年3月24日 11:25

Happened to me too, try to "Purge All Caches" on Wordpress settings/hosting service provider.

Then to check, open a browser in "Incognito mode" (chrome, make sure you didn't have any Incognito windows open already, otherwise you'll have to close them all and re-open a new Incognito window), then head over to your www.mydomain.com/robots.txt You should now be able to see the updated content file like expected.

Dave Ross · Accepted Answer · 2015年12月9日 13:45

A WordPress site doesn't need an actual robots.txt file. WordPress itself will intercept requests for robots.txt and respond with its own version, and that's what your plugin is most likely trying to modify.

Delete or rename your actual robots.txt file and try it.

Nikhil · Accepted Answer · 2015年12月9日 09:17

Once I experienced the same issue, this is what I did to fix the issue.

Edit the robots.txt file directly (using FTP/SSH),

User-agent: * 
Disallow: /wp-admin/ 
Disallow: /wp-includes/

There are two reasons if the robots files not updated when you edited using a plugin.

File permission.
Some other plugin is reverting the changes.

Also try to update the search engine visibility settings (this might not be an issue, but just have a look).

Settings -> Reading -> Search Engine Visibility (Uncheck this option) -> Save

I hope this helps.

Robots.txt not updating

About