Why We Use robot.txt file In Seo..?
 

Hello friends,

Why We Use robot.txt file In Seo..?

When a search engine crawler comes to your site, it will look for a special file on your site. That file is called robots.txt and it tells the search engine spider, which Web pages of your site should be indexed and which Web pages should be ignored.

The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.

Robots.txt is a text file you put on your site to tell search robots which pages you wouls like them not to visit

Robots.txt is a text file you put on your site to tell search robots which pages you wouls like them not to visit

In a nutshell. Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit.

You can use Google Search Consol to test your sites robots.txt. I definitely recommend doing this to make sure it is working properly