Site Owners Forums - Webmaster Forums - View Single Post

festivlbuts · 03-29-2013, 12:34 AM

A robots.txt file on a website will function as a request that specified robots ignore specified files or directories when crawling a site.

03-29-2013, 12:34 AM	#3
festivlbuts Registered User Join Date: Mar 2013 Posts: 3	A robots.txt file on a website will function as a request that specified robots ignore specified files or directories when crawling a site.