|
Hello
Robots.txt is the file which we used to block the content from crawler so that search engine will not crawled and index our data which we mention in robots file.
Mainly the confidential data which we want to hide block with the help of robots file.
Hope you understand.
|