Site Owners Forums - Webmaster Forums - View Single Post

titly555 · 11-26-2012, 05:26 AM

The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) on how to crawl & index pages on their website.

11-26-2012, 05:26 AM	#2
titly555 Registered User Join Date: Nov 2012 Location: India Posts: 109	The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) on how to crawl & index pages on their website.