Robots.txt are supposed to give the web crawlers instructions

Robots.txt file is a text file. That stops web crawler software such as Googlebot. from crawling certain pages of your site. The file is essentially a list of commands, such Allow and Disallow , that tell web crawlers which URLs they can or cannot retrieve.

Cái này hay thật đó, cảm ơn vì chia sẻ của bạn