[DewangJani]

Robots.txt is a text file webmasters create to instruct robots (typically search engine robots) on how to crawl & index pages on their website. Robots. text is by no means mandatory for search engines but general search engines obey what they are asked not to do. The location of Robots. The text is very important. It must be in the main directory because otherwise user agents (search engines) will not be able to find it – they do not search the whole site for a file named robots.txt.