what is the use of robot.txt? - Site Owners Forums

europeanoversea · 03-18-2016, 01:21 AM

what is the use of robot.txt?

alicemark15 · 03-18-2016, 01:52 AM

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.

arianagrand · 03-18-2016, 01:58 AM

robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.

daviddakarai · 03-18-2016, 03:00 AM

Robots.txt is a simple text file that is placed on your site’s root directory. It is that file on your website that tells these search engine robots what to crawl and what not to crawl on your site. To facilitate auto-discovery of your sitemap file through your robots.txt, all you have to do is place a directive with the URL in your robots.txt

krify · 03-18-2016, 03:17 AM

The main use of Robot text file is to make search engines know what is are the URLs need to cowl around the websites !

robot dot text is used to upload at the time of site creation then site with sitemap upload using ftp publications.

AshokDixit89 · 03-18-2016, 06:00 AM

Use of Robots.txt - The most common usage of Robots.txt is to ban crawlers from visiting private folders or content that gives them no additional information.

Robots.txt Allowing Access to Specific Crawlers.
Allow everything apart from certain patterns of URLs.

indianlawwatch · 03-18-2016, 11:20 PM

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.

Sunsite · 03-19-2016, 01:24 AM

It is a instruction to robots in a text how to crawl and index pages on their website

Sangeetakashyap · 03-19-2016, 04:14 AM

Robot.txt is a text file that has been used by website owners to give instructions to the web robots about the website. It helps the robots of Google to crawl each page of website so that it can index it properly & can rank it well in the Google search engine easily. It is an important factor of On Page of any website.

accutranskpo · 03-22-2016, 04:37 AM

robots.txt is a text file which is used by web-master to give crawling instructions to the search engine crawlers which page is to be crawl and which page is not.

vashikaransutra · 03-22-2016, 04:54 AM

It is must file for every website if webmasters want to do SEO. This file tells to Robots about which files should crawl or which not. It must place on root of the website.

Nexevo Technolo · 03-22-2016, 05:49 AM

When a search engine crawler comes to your site, it will look for a special file on your site. That file is called robots.txt and it tells the search engine spider, which Web pages of your site should be indexed and which Web pages should be ignored

aajokhio · 03-22-2016, 04:40 PM

Basically, Robots Txt File Allow or Disallow with Google Robots for Crawl your Post, Pages etc. You set it as your requirement after that Google Robots crawl your post pages you have set up in Robots Txt File.

SanGoku · 03-22-2016, 08:31 PM

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

hap53144 · 03-23-2016, 08:58 AM

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sen sitive data, it is too naïve to rely on robots.txt to protect it from being indexed and displayed in search results.
Thanks

03-18-2016, 01:21 AM	#1
europeanoversea Registered User Join Date: Jan 2016 Posts: 24	what is the use of robot.txt? what is the use of robot.txt? __________________ Singapore study visa cost \| Singapore study visa singapore tourist visa consultants punjab singapore tourist visa agents

03-18-2016, 01:52 AM	#2
alicemark15 Registered User Join Date: Oct 2015 Posts: 95	Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. __________________ SEO Project Management Tool

03-18-2016, 01:58 AM	#3
arianagrand Registered User Join Date: Feb 2014 Posts: 197	robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website. __________________ Team Building Organisers in Delhi \| Corporate Team Building Activities \| Newspaper Ad Agency in Delhi \| Newspaper Advertising

03-18-2016, 03:00 AM	#4
daviddakarai Registered User Join Date: Jan 2015 Posts: 677	Robots.txt is a simple text file that is placed on your site’s root directory. It is that file on your website that tells these search engine robots what to crawl and what not to crawl on your site. To facilitate auto-discovery of your sitemap file through your robots.txt, all you have to do is place a directive with the URL in your robots.txt __________________ Web Designing Comapny , Web Designing Company in Hyderabad , SEO services in Hyderabad , Mobile Apps Development Company in Hyderabad

03-18-2016, 03:17 AM	#5
krify Registered User Join Date: Feb 2016 Posts: 83	The main use of Robot text file is to make search engines know what is are the URLs need to cowl around the websites ! robot dot text is used to upload at the time of site creation then site with sitemap upload using ftp publications. __________________ Mobile, Web Apps Development Company \| Extended Services in Digital Marketing KRIFY DOT CO

03-18-2016, 06:00 AM	#6
AshokDixit89 Registered User Join Date: Jan 2016 Location: Mumbai, India Posts: 1,064	Use of Robots.txt - The most common usage of Robots.txt is to ban crawlers from visiting private folders or content that gives them no additional information. Robots.txt Allowing Access to Specific Crawlers. Allow everything apart from certain patterns of URLs. __________________ Screener\|Equity Research

03-18-2016, 11:20 PM	#7
indianlawwatch Registered User Join Date: Feb 2016 Posts: 44	Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. __________________ indian lawyers forum recent supreme court decisions the constitution of india alternative dispute resolution Consumer law in india law firms in delhi

03-19-2016, 01:24 AM	#8
Sunsite Registered User Join Date: Mar 2016 Posts: 8	It is a instruction to robots in a text how to crawl and index pages on their website __________________ business onlinehttp://bizviewpoint.blogspot.com

03-19-2016, 04:14 AM	#9
Sangeetakashyap Registered User Join Date: Sep 2015 Location: Delhi Posts: 145	Robot.txt is a text file that has been used by website owners to give instructions to the web robots about the website. It helps the robots of Google to crawl each page of website so that it can index it properly & can rank it well in the Google search engine easily. It is an important factor of On Page of any website. __________________ Bike On Rent Free Rental Classified Site Dress On Rent

03-22-2016, 04:37 AM	#10
accutranskpo Registered User Join Date: Mar 2016 Posts: 15	robots.txt is a text file which is used by web-master to give crawling instructions to the search engine crawlers which page is to be crawl and which page is not.

03-22-2016, 04:54 AM	#11
vashikaransutra Registered User Join Date: Aug 2015 Posts: 279	It is must file for every website if webmasters want to do SEO. This file tells to Robots about which files should crawl or which not. It must place on root of the website. __________________ Love Problem Solution Baba\|Get Love Back By Vashikaran\|Mohini Vashikaran Mantra\|Love Guru In India\| FAMOUS PANDIT

03-22-2016, 05:49 AM	#12
Nexevo Technolo Registered User Join Date: Sep 2015 Location: Bangalore Posts: 548	When a search engine crawler comes to your site, it will look for a special file on your site. That file is called robots.txt and it tells the search engine spider, which Web pages of your site should be indexed and which Web pages should be ignored __________________ Website Development Company in Bangalore \| Website Development in Bangalore \| Website Development Companies in Bangalore \| Website Design Company Bangalore \| Website Design Company Bangalore

03-22-2016, 04:40 PM	#13
aajokhio Registered User Join Date: May 2015 Location: Hyderabad/Pakistan Posts: 25	Basically, Robots Txt File Allow or Disallow with Google Robots for Crawl your Post, Pages etc. You set it as your requirement after that Google Robots crawl your post pages you have set up in Robots Txt File.

03-22-2016, 08:31 PM	#14
SanGoku Registered User Join Date: Apr 2013 Posts: 382	Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

03-23-2016, 08:58 AM	#15
hap53144 Registered User Join Date: Feb 2016 Posts: 34	Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sen sitive data, it is too naïve to rely on robots.txt to protect it from being indexed and displayed in search results. Thanks __________________ Create Any Stunning Animated Graphics with https://goo.gl/VTAJKu In Just A Few Clicks \| New trick works in https://goo.gl/QLv7VQ for ANY Facebook Ad \| Turns Your Videos Into Sales Machine at https://goo.gl/BTgdrk

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
what is robot.txt?	rubysethi	Search Engine Optimization	0	12-19-2014 11:27 PM
How to know about robot.txt ?	vanaja	Search Engine Optimization	10	10-08-2014 04:39 AM
how can Create the Robot.txt	pankaj	Search Engine Optimization	3	10-08-2014 03:43 AM
how can create the Robot.txt file	eagleeye dental	Search Engine Optimization	0	09-01-2014 09:40 PM