What is robots.txt? - Site Owners Forums

seoinheritx · 11-26-2012, 12:01 AM

Robots.txt is a text file used to give instructions to the search engine crawlers about the caching and indexing of a webpage, domain, directory or a file of a website.

titly555 · 11-26-2012, 05:26 AM

The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) on how to crawl & index pages on their website.

mytimeandmoneys · 11-26-2012, 07:42 AM

Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.

IAMalfin · 11-26-2012, 10:02 PM

Robots.txt is a convention to prevent cooperating web crawlers and other web robots from accessing all or part of a website which is otherwise publicly viewable.

sunitisood16 · 11-27-2012, 02:58 AM

Robots.txt files inform search engine spiders how to interact with indexing your content. If you do not have a robots.txt file, your server logs will return 404 errors whenever a bot tries to access your robots.txt file.

outure11 · 11-27-2012, 04:56 AM

This is very useful post. Thank you.........

john mathew · 11-28-2012, 05:09 AM

Thanks for share this information.

abhirampathak3 · 11-28-2012, 10:53 PM

The robots.txt is a simple text file in your web site that inform search engine bots how to crawl and index website or web pages.

arindamdutta16 · 11-30-2012, 03:56 AM

The robots.txt is a simple text file in your web site that informs search engine bots how to crawl and index website or web pages. It is great when search engines frequently visit your site and index your content but often there are cases when indexing parts of your online content is not what you want.

likit · 12-03-2012, 05:46 AM

Robots.txt is text file that holding the information for search engine crawler about caching and indexing of webpage, Domain..etc

laceywilliams12 · 12-04-2012, 11:07 PM

Robots.txt is a text file and having the information for dofollow or no-follow for the crawlers to crawl a website web pages, and providing them indexing.

Eminenz Cw · 12-05-2012, 04:28 AM

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note �Please, do not enter� on an unlocked door � e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter.

muzz · 12-05-2012, 06:24 AM

robot.txt tells the search engine which data you want to crawl and which is not.

stephan07 · 12-11-2012, 03:43 AM

Robot.txt is a simple notepad file through which crawler crawl your website.

ronaldotson45 · 12-11-2012, 11:53 PM

Robot.txt file indicate search crawler which information in this website is prohibited or do not crawl. if owner of the site does not want any of the file to be indexed or crawled by robot or spider then robot.txt file being used

11-26-2012, 12:01 AM	#1
seoinheritx Registered User Join Date: Dec 2011 Posts: 48	What is robots.txt? Robots.txt is a text file used to give instructions to the search engine crawlers about the caching and indexing of a webpage, domain, directory or a file of a website. __________________ Asp.net Application development \| Sharepoint Development

11-26-2012, 10:02 PM	#4
IAMalfin Registered User Join Date: Jun 2011 Posts: 325	Robots.txt is a convention to prevent cooperating web crawlers and other web robots from accessing all or part of a website which is otherwise publicly viewable. __________________ Outsourcing in the Philippines Virtual Staff Outsourcing Accounts Payable

11-27-2012, 04:56 AM	#6
outure11 Registered User Join Date: Oct 2012 Posts: 334	This is very useful post. Thank you......... __________________ plus size bra

11-28-2012, 05:09 AM	#7
john mathew Registered User Join Date: Feb 2012 Posts: 225	Thanks for share this information. __________________ fixed term loan \| apply for a home loan

11-28-2012, 10:53 PM	#8
abhirampathak3 Registered User Join Date: Nov 2012 Location: India Posts: 103	The robots.txt is a simple text file in your web site that inform search engine bots how to crawl and index website or web pages. __________________ Web Development Services - Shopping Cart - PrestaShop, Zencart; CMS - WordPress, Joomla; Custom Development Quality SEO Services - SEO, Pay Per Click (PPC), SEM, Link Popularity Building, SEO Copywriting, SMO, SMM

11-26-2012, 05:26 AM	#2
titly555 Registered User Join Date: Nov 2012 Location: India Posts: 109	The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) on how to crawl & index pages on their website.

11-26-2012, 07:42 AM	#3
mytimeandmoneys Registered User Join Date: Nov 2012 Posts: 5	Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.

11-27-2012, 02:58 AM	#5
sunitisood16 Registered User Join Date: Nov 2012 Posts: 76	Robots.txt files inform search engine spiders how to interact with indexing your content. If you do not have a robots.txt file, your server logs will return 404 errors whenever a bot tries to access your robots.txt file.

11-30-2012, 03:56 AM	#9
arindamdutta16 Registered User Join Date: Nov 2012 Location: India Posts: 103	The robots.txt is a simple text file in your web site that informs search engine bots how to crawl and index website or web pages. It is great when search engines frequently visit your site and index your content but often there are cases when indexing parts of your online content is not what you want. __________________ Best Web Design Services - Corporate Web Design, Creative Design, Custom Web Design, Website Re-Design Quality SEO Services - - SEO, Pay Per Click (PPC), SEM, Link Popularity Building, SEO Copywriting, SMO, SMM

12-03-2012, 05:46 AM	#10
likit Registered User Join Date: Oct 2012 Posts: 149	Robots.txt is text file that holding the information for search engine crawler about caching and indexing of webpage, Domain..etc

12-04-2012, 11:07 PM	#11
laceywilliams12 Registered User Join Date: Nov 2012 Posts: 71	Robots.txt is a text file and having the information for dofollow or no-follow for the crawlers to crawl a website web pages, and providing them indexing.

12-05-2012, 04:28 AM	#12
Eminenz Cw Registered User Join Date: Oct 2012 Posts: 138	Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note �Please, do not enter� on an unlocked door � e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. __________________ Rayban Sunglasses \| Spa \| Milk Powder \| kids theatre \| Dentist in Delhi \| Leh Ladakh Packages

12-05-2012, 06:24 AM	#13
muzz Registered User Join Date: Sep 2012 Posts: 196	robot.txt tells the search engine which data you want to crawl and which is not. __________________ executive suites vancouver extended stay vancouver monthly rentals Vancouver

12-11-2012, 03:43 AM	#14
stephan07 Registered User Join Date: Dec 2012 Posts: 25	Robot.txt is a simple notepad file through which crawler crawl your website.

12-11-2012, 11:53 PM	#15
ronaldotson45 Registered User Join Date: Dec 2012 Posts: 5	Robot.txt file indicate search crawler which information in this website is prohibited or do not crawl. if owner of the site does not want any of the file to be indexed or crawled by robot or spider then robot.txt file being used

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Evil bots consuming all my sites traffic even after i disallowed them in robots.txt	Fking	General Discussion	1	12-13-2013 11:27 PM
Robots.txt	sameerseo	Search Engine Optimization	26	09-28-2012 04:18 AM
robots.txt	wheelbarrow001	Google	2	07-27-2012 04:10 AM
Unable to delete robots.txt file	notfake	Yahoo	0	04-29-2012 02:19 AM
What Is Robots.txt?	samlko	Search Engine Optimization	13	03-09-2012 02:37 PM