|
It is a text file which instructs search engine spiders or crawlers on what to do. It tells specific web spiders on which specific web pages to index. Robots are configured to read text.It contains restrictions for Web Spiders, telling them where they have permission to search. It is like defining rules for search engine spiders (robots) what to follow and what not to.
|