Robots.txt is a text file which has few lines and this file interacts with all kind of crawlers or spiders like Googlebot, which is Google search engine's spider.

Search Engine always want to index good and fresh contain on the web and these robot.txt instruct the web crawlers to how to index and crawl your blog in the search results.

Remember,Crawler or spider will firstly look at your robots.txt files to obey the rules you have instructed It's mean you can restrict any web page on your blog from web crawlers so that it can’t get indexed in search engines like your blog your demo page, labels page, Achieve or any other pages that are not as important to get indexed.

Robots.txt is a text file which has few lines and this file interacts with all kind of crawlers or spiders like Googlebot, which is Google search engine's spider.

Search Engine always want to index good and fresh contain on the web and these robot.txt instruct the web crawlers to how to index and crawl your blog in the search results.