What is robot.txt file and what are the benefits of using it? - Page 2 - Site Owners Forums

henrysmithh71 · 11-28-2016, 11:26 PM

Robots.txt file is used to give instruction to the search engine crawler to which page has to crawled or not by allowing or disallowing.

georgewills · 11-28-2016, 11:40 PM

The file tells the search engine crawlers which parts to web and which parts to leave alone in a website, differing between what is viewable to the public and what is viewable to the creators of the website alone.

stuartsmithh5 · 11-28-2016, 11:45 PM

Robots.txt file is used to give instruction to the search engine crawler to which page has to crawled or not by allowing or disallowing.

pattroderick · 11-28-2016, 11:51 PM

Robots.txt file is for giving instruction to the crawlers indicates which one to crawl and which are don't crawl.

Dentcare · 11-29-2016, 04:10 AM

Robot.txt used to provide guidance to the search engine crawler about the web pages that which page to be crawled and which is not.

luffy268 · 12-14-2016, 12:43 AM

The location of robots.txt is very important. It must be in the main directory because otherwise user agents (search engines) will not be able to find it – they do not search the whole site for a file named robots.txt. Instead, they look first in the main directory and if they don't find it there, they simply assume that this site does not have a robots.txt file and therefore they index everything they find along the way. So, if you don't put robots.txt in the right place, do not be surprised that search engines index your whole site.

ChrisRogers123 · 12-14-2016, 02:44 AM

It is a type of text file which allows search engine to what to crawl or what to not

Pandith Raghave · 12-14-2016, 03:13 AM

Robots.txt is document you put on your site to tell look robots which pages you might want them Do and Don't to visit.

jeffmccoy · 12-14-2016, 04:14 AM

Robots.txt file is a file through which we give instructions to the search engines whether to crawl or not the particular web page.

stuartsmithh5 · 12-15-2016, 12:54 AM

Robot.txt file is used to provide instruction to the search engine crawler about the webpages that which page to be crawled and which are not to be crawled.

Markco92 · 12-15-2016, 03:35 AM

Robot.txt file is used to provide instruction to the search engine crawler about the webpages that which page to be crawled and which not.

FerinKings · 12-16-2016, 12:23 AM

robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.

wiliamjamesh · 12-17-2016, 01:49 AM

The purpose of a robots.txt file is to tell search engines not to crawl the contents of specific directories or even individual pages within a website. ... However there are a couple of reasons why all websites would benefit from having one. The first thing search engines look for is the robots.txt file.

georgewills · 12-17-2016, 05:22 AM

Robots.txt file is used to give instruction to the search engine crawler to which page has to crawled or not by allowing or disallowing.

LukeBarter · 12-19-2016, 01:44 AM

It is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.

11-28-2016, 11:26 PM	#16
henrysmithh71 Registered User Join Date: Sep 2016 Posts: 75	Robots.txt file is used to give instruction to the search engine crawler to which page has to crawled or not by allowing or disallowing. __________________ Job vacancies \| Freshers world \| Walkin Interviews

11-28-2016, 11:40 PM	#17
georgewills Registered User Join Date: Sep 2016 Posts: 70	The file tells the search engine crawlers which parts to web and which parts to leave alone in a website, differing between what is viewable to the public and what is viewable to the creators of the website alone. __________________ Apartments In Israels \| Apartments For Rent In Israel \| Holidays In Israel

11-28-2016, 11:45 PM	#18
stuartsmithh5 Registered User Join Date: Oct 2016 Posts: 81	Robots.txt file is used to give instruction to the search engine crawler to which page has to crawled or not by allowing or disallowing. __________________ Big Party Tents \| Discount Party Tents \| PVC Party Tents

11-28-2016, 11:51 PM	#19
pattroderick Registered User Join Date: Jul 2016 Posts: 243	Robots.txt file is for giving instruction to the crawlers indicates which one to crawl and which are don't crawl. __________________ Outdoor Patio Umbrellas \| patio umbrellas

11-29-2016, 04:10 AM	#20
Dentcare Registered User Join Date: Aug 2016 Posts: 187	Robot.txt used to provide guidance to the search engine crawler about the web pages that which page to be crawled and which is not. __________________ Best Braces Treatment In Chennai \| Dental Clinic In Besant Nagar \| Best Dental Clinic In Besant \| Dental Clinic In Thiruvanmiyur \| Best Dental Clinic In Thiruvanmiyur \| Dental Clinic In Mylapore \| Best Dental Clinic In Mylapore

12-14-2016, 12:43 AM	#21
luffy268 Registered User Join Date: Dec 2016 Posts: 94	The location of robots.txt is very important. It must be in the main directory because otherwise user agents (search engines) will not be able to find it – they do not search the whole site for a file named robots.txt. Instead, they look first in the main directory and if they don't find it there, they simply assume that this site does not have a robots.txt file and therefore they index everything they find along the way. So, if you don't put robots.txt in the right place, do not be surprised that search engines index your whole site. __________________ *hakireview\|hakireview.com\|http://hakireview.com*

12-14-2016, 02:44 AM	#22
ChrisRogers123 Registered User Join Date: Aug 2016 Location: USA Posts: 117	It is a type of text file which allows search engine to what to crawl or what to not __________________ Benefits Of SEO Company \|\| Benefits Of Social Media Marketing

12-14-2016, 03:13 AM	#23
Pandith Raghave Registered User Join Date: Aug 2016 Location: London Posts: 115	Robots.txt is document you put on your site to tell look robots which pages you might want them Do and Don't to visit. __________________ Famous Astrologer in London \| World Famous Astrologer in London \| Top Astrology Services in London \| Indian Best Astrologer in London \| Indian Astrologers in London UK \| Top Astrologer London \| Top Indian Astrologer London

12-14-2016, 04:14 AM	#24
jeffmccoy Registered User Join Date: Feb 2016 Location: Green Cove Springs, FL Posts: 321	Robots.txt file is a file through which we give instructions to the search engines whether to crawl or not the particular web page. __________________ Baby Shower Gifts Best Baby Gift

12-15-2016, 12:54 AM	#25
stuartsmithh5 Registered User Join Date: Oct 2016 Posts: 81	Robot.txt file is used to provide instruction to the search engine crawler about the webpages that which page to be crawled and which are not to be crawled. __________________ Big Party Tents \| Discount Party Tents \| PVC Party Tents

12-15-2016, 03:35 AM	#26
Markco92 Registered User Join Date: Dec 2016 Location: New York Posts: 113	Robot.txt file is used to provide instruction to the search engine crawler about the webpages that which page to be crawled and which not. __________________ Claim ME now anytime to get FREE BONUS WP Site Guardian 2017 Reviews\|\|WP Site Guardian 2017 Bonus ReachMultiply ELITE Lifetime \|\| WP Toolkit Review Social Media Firesale Review\|\|WP Domination Secrets Review Official Page

12-16-2016, 12:23 AM	#27
FerinKings Registered User Join Date: Aug 2016 Posts: 205	robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website. __________________ *Common Core Math* \| *Best Math Tutors* \| *Stem Program*

12-17-2016, 01:49 AM	#28
wiliamjamesh Registered User Join Date: Sep 2016 Posts: 617	The purpose of a robots.txt file is to tell search engines not to crawl the contents of specific directories or even individual pages within a website. ... However there are a couple of reasons why all websites would benefit from having one. The first thing search engines look for is the robots.txt file. __________________ Digital Transformation Expert \| Digital Transformation Company \| Digital transformation service providers

12-17-2016, 05:22 AM	#29
georgewills Registered User Join Date: Sep 2016 Posts: 70	Robots.txt file is used to give instruction to the search engine crawler to which page has to crawled or not by allowing or disallowing. __________________ Apartments In Israels \| Apartments For Rent In Israel \| Holidays In Israel

12-19-2016, 01:44 AM	#30
LukeBarter Registered User Join Date: Sep 2016 Posts: 171	It is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website. __________________ *Exchange Second Hand Goods* \|\| *Free Classifieds in India*

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)