Site Owners Forums - Webmaster Forums - View Single Post

champion · 02-24-2013, 10:20 PM

Robots.txt is a file through which you can guide search engines to crawl or not to crawl certain sections of your website.Robot.txt is used to prevent crawlers crawl on a certain page.

02-24-2013, 10:20 PM	#2
champion Registered User Join Date: Feb 2013 Posts: 21	Robots.txt is a file through which you can guide search engines to crawl or not to crawl certain sections of your website.Robot.txt is used to prevent crawlers crawl on a certain page.