robots.txt file is used for control search engine bot.

Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit.

robots.txt is a text file webmasters create to instruct robots how to crawl and index pages on their website.

The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.

Use of Robots.txt - The most common usage of Robots.txt is to ban crawlers from visiting private folders or content that gives them no additional information.

Robots.txt Allowing Access to Specific Crawlers.
Allow everything apart from certain patterns of URLs.

Robots.txt file give instruction to search engine spiders, how search engine spider to see and interact with your web pages.

A robots. txt file tells search engine crawlers which URLs the crawler can access on your site. This is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google.