Site Owners Forums - Webmaster Forums - View Single Post

andersonsjohn · 03-08-2012, 02:52 AM

Robots.txt will tell the search engine what to crawl and what not to. This helps if you do not want search engines to crawl and index some of your private data.

03-08-2012, 02:52 AM	#7
andersonsjohn Registered User Join Date: Jan 2012 Posts: 177	Robots.txt will tell the search engine what to crawl and what not to. This helps if you do not want search engines to crawl and index some of your private data.