robots.txt
 

The robots.txt is a stored in the root directory of the web site under ASCII coding text files, it usually against Internet search led also called web spiders, which of the site should not be content is search engine for the rovers, which can be obtained

Hello

Robots.txt is the file which we used to block the content from crawler so that search engine will not crawled and index our data which we mention in robots file.

Mainly the confidential data which we want to hide block with the help of robots file.


Hope you understand.

Yes robot.txt is helpful to block crawler to crawl your website. In this your mention that area where you dont want to visit crawler.