View Single Post
Old 06-03-2013, 04:03 AM   #16
DewangJani
Registered User
 
Join Date: Apr 2013
Location: India
Posts: 9
Robots.txt is a text file webmasters create to instruct robots (typically search engine robots) on how to crawl & index pages on their website. Robots. text is by no means mandatory for search engines but general search engines obey what they are asked not to do. The location of Robots. The text is very important. It must be in the main directory because otherwise user agents (search engines) will not be able to find it – they do not search the whole site for a file named robots.txt.

Last edited by DewangJani; 07-20-2014 at 11:27 PM..
DewangJani is offline   Reply With Quote