What is the meaning of User-agent and Disallow in robots.txt file in wordpress
What is the meaning of User-agent and Disallow in robots.txt file in WordPress. I am little bit confused in this.
Topic robots.txt Wordpress
Category Web
User-agent
They are Web Robots (also known as Web Wanderers, Crawlers, or Spiders), are programs that traverse the Web automatically. Search engines such as Google use them to index the web content, spammers use them to scan for email addresses, and they have many other uses.
Disallow
It tells the robots that it should not visit the pages or directory of site which are not allowed by this tag.
About
Geeks Mental is a community that publishes articles and tutorials about Web, Android, Data Science, new techniques and Linux security.