Site Owners Forums - Webmaster Forums - View Single Post

bestindiatours · 10-14-2014, 12:12 AM

its the text file which with the name robot.txt use to disallow the crawler to crawl the page that we don't need to crawl

10-14-2014, 12:12 AM	#15
bestindiatours Registered User Join Date: Oct 2014 Posts: 5	its the text file which with the name robot.txt use to disallow the crawler to crawl the page that we don't need to crawl