What Is Robots.txt? - Site Owners Forums

samlko · 03-06-2012, 03:35 AM

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit.

josefaryan · 03-06-2012, 03:46 AM

Robots.txt file can be used to tell search engines that which of your website pages/directories should not be crawled.

blueapple · 03-06-2012, 03:52 AM

Somehow you can say like this but purpose is same to disallow search engine to index the content.

amitmax · 03-06-2012, 04:06 AM

Very true.Generally we use it to save unusual crawling of spider on those pages having duplicate content or files which we don't want to share publicly.
Thanks

Johnmark · 03-07-2012, 04:28 AM

it gives the instruction to not crawl the specific pages or file mention in txt

jessysmith · 03-08-2012, 01:13 AM

Robots.txt will inform the search engine what are the pages should be crawled

andersonsjohn · 03-08-2012, 02:52 AM

Robots.txt will tell the search engine what to crawl and what not to. This helps if you do not want search engines to crawl and index some of your private data.

froz · 03-08-2012, 03:54 AM

Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site..

jessica733 · 03-08-2012, 07:13 AM

Thanks to share this great information with us, Robot.txt files are very important in on-page field to make our website error free.

wheelbarrow001 · 03-08-2012, 11:30 PM

The robots.txt is for a search engine, visit the site of the time want to see the first file. The robots.txt file told spiders in the server is what document can be looked at.
When a search spider visit a website, it will first check the site to root directory whether there is the robots.txt, if present, search according to the document robot to determine the content of the scope of the visit; If the file does not exist, all the search spiders will be able to access web site was not all password protected pages.

atoment01 · 03-08-2012, 11:40 PM

robot.txt file is simple text document.
1. open notepad
2. type these lines.
3. save your file as robots.txt (extension should be .txt)
4. placed this file in your root folder
5. you can view it

Diane Forster · 03-09-2012, 03:33 AM

It is a code within the root file that acts to avoid spiders from crawling over the site or page. It is a technique used in SEO to avoid indexing.

harryjohn99 · 03-09-2012, 04:04 AM

Robot.txt is the way to stop the search engines to crawl any page of the website. This is required because some time there is private data on some pages of the website and the website owner does not want search engines to crawl those pages and read that useful information. In those cases robot.txt is the file in which all the pages that we don't want to crawl are added and this file is then added to the root folder of the website.

electra · 03-09-2012, 02:37 PM

Robots.txt is a file through which you can guide search engines to crawl or not to crawl certain sections of your website.

Thanks and Regards

03-06-2012, 03:35 AM	#1
samlko Registered User Join Date: Sep 2009 Posts: 67	What Is Robots.txt? Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

03-07-2012, 04:28 AM	#5
Johnmark Registered User Join Date: Dec 2011 Posts: 153	it gives the instruction to not crawl the specific pages or file mention in txt __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

03-08-2012, 01:13 AM	#6
jessysmith Registered User Join Date: Feb 2012 Posts: 27	Robots.txt will inform the search engine what are the pages should be crawled __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

03-08-2012, 03:54 AM	#8
froz Registered User Join Date: Feb 2012 Posts: 86	Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site.. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

03-08-2012, 07:13 AM	#9
jessica733 Registered User Join Date: Jul 2011 Location: Nicosia, Cyprus Posts: 57	Thanks to share this great information with us, Robot.txt files are very important in on-page field to make our website error free. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

03-06-2012, 03:46 AM	#2
josefaryan Registered User Join Date: Mar 2012 Posts: 259	Robots.txt file can be used to tell search engines that which of your website pages/directories should not be crawled.

03-06-2012, 03:52 AM	#3
blueapple Registered User Join Date: Feb 2012 Posts: 92	Somehow you can say like this but purpose is same to disallow search engine to index the content.

03-06-2012, 04:06 AM	#4
amitmax Registered User Join Date: Aug 2010 Posts: 215	Very true.Generally we use it to save unusual crawling of spider on those pages having duplicate content or files which we don't want to share publicly. Thanks

03-08-2012, 02:52 AM	#7
andersonsjohn Registered User Join Date: Jan 2012 Posts: 177	Robots.txt will tell the search engine what to crawl and what not to. This helps if you do not want search engines to crawl and index some of your private data.

03-08-2012, 11:30 PM	#10
wheelbarrow001 Registered User Join Date: Nov 2011 Posts: 35	The robots.txt is for a search engine, visit the site of the time want to see the first file. The robots.txt file told spiders in the server is what document can be looked at. When a search spider visit a website, it will first check the site to root directory whether there is the robots.txt, if present, search according to the document robot to determine the content of the scope of the visit; If the file does not exist, all the search spiders will be able to access web site was not all password protected pages. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

03-08-2012, 11:40 PM	#11
atoment01 Registered User Join Date: Mar 2012 Posts: 15	robot.txt file is simple text document. 1. open notepad 2. type these lines. 3. save your file as robots.txt (extension should be .txt) 4. placed this file in your root folder 5. you can view it

03-09-2012, 03:33 AM	#12
Diane Forster Registered User Join Date: Aug 2011 Posts: 290	It is a code within the root file that acts to avoid spiders from crawling over the site or page. It is a technique used in SEO to avoid indexing. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. Junkies

03-09-2012, 04:04 AM	#13
harryjohn99 Registered User Join Date: Nov 2011 Posts: 51	Robot.txt is the way to stop the search engines to crawl any page of the website. This is required because some time there is private data on some pages of the website and the website owner does not want search engines to crawl those pages and read that useful information. In those cases robot.txt is the file in which all the pages that we don't want to crawl are added and this file is then added to the root folder of the website. Last edited by harryjohn99; 01-08-2014 at 12:44 AM..

03-09-2012, 02:37 PM	#14
electra Registered User Join Date: Mar 2012 Posts: 44	Robots.txt is a file through which you can guide search engines to crawl or not to crawl certain sections of your website. Thanks and Regards __________________ Need webhosting now? Read To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. or for business hosting read To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. . If You like, please also visit To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. to get some discount. Thank you

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
sitemap.xml and robots.txt?	jamesranatte	Search Engine Optimization	5	01-31-2012 11:16 PM