Site Owners Forums - Webmaster Forums - View Single Post

sunitisood16 · 11-27-2012, 02:58 AM

Robots.txt files inform search engine spiders how to interact with indexing your content. If you do not have a robots.txt file, your server logs will return 404 errors whenever a bot tries to access your robots.txt file.

11-27-2012, 02:58 AM	#5
sunitisood16 Registered User Join Date: Nov 2012 Posts: 76	Robots.txt files inform search engine spiders how to interact with indexing your content. If you do not have a robots.txt file, your server logs will return 404 errors whenever a bot tries to access your robots.txt file.