What is robot.txt file and what are the benefits of using it? - Site Owners Forums

Martinricky · 03-05-2016, 01:39 AM

Hello,

What is robot.txt file and what are the benefits of using it?

Thank you

aditya12 · 03-05-2016, 04:23 AM

Robot.txt file is used to provide instruction to the search engine crawler about the webpages that which page to be crawled and which not.

Best schools in Chandigarh Best schools in Mohali Best schools in Panchkula Best schools in Tricity Colleges in Chandigarh Colleges in Mohali Colleges in Tricity Colleges in Panchkula

tenso500 · 03-05-2016, 04:52 AM

The file tells the search engine crawlers which parts to web and which parts to leave alone in a website, differing between what is viewable to the public and what is viewable to the creators of the website alone.

Admyrin · 03-05-2016, 05:17 AM

Robot.txt file is used to provide instruction to the search engine crawler about the webpages that which page to be crawled and which not.

langtu1292 · 03-24-2016, 06:46 AM

you can search on google..many thing for you

josechukkiri · 03-24-2016, 07:14 AM

The explanation about robot.txt files are satisfactory to me and thanks to the concerned.

sudeepkhana · 03-25-2016, 05:57 AM

Robot.txt is a text file kept on the web servers. The robot.txt allows the site admin to permit/allow which Search engines(like google, yahoo,bing) can crawl the site. Some times the admins won't allow any of the search engines to crawl at all.

Devin Mataka · 03-25-2016, 06:06 AM

The robots.txt file is a simple text file (no html) that is placed in your website’s root directory in order to tell the search engines which pages to index and which to skip. Many webmasters utilize this file to help the search engines index the content of their websites.
If webmasters can tell the search engine spiders to skip pages that they do not consider important enough to be crawled (eg. printable versions of pages, .pdf files etc.), then they have a better opportunity to have their most valuable pages featured in the search engine results pages.The robots.txt file is a simple method of essentially easing the process for the spiders to return the most relevant search results.
Another benefit of having a robots.txt is that you can specify the location of the Google .xml or Yahoo sitemap. This also increases spiderability for the search engines.

rahul3214 · 04-07-2016, 09:42 PM

Robots.txt file is important for seo, search engine spiders first see that where your robots.txt file because they do not know which file crawl or which not. In robots.txt file google know which file crawl.

karanjain · 04-08-2016, 12:27 PM

nice and awesome
IPL 9 - Indian Premier League 2016
IPL Auction 2016 Live Results Shane Watson And Yuvraj Sold On Highest Bidding

albertocosta · 04-09-2016, 04:57 AM

The robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.

brknny · 04-10-2016, 11:27 PM

Robots.txt file is a file through which we give instructions to the search engines whether to crawl or not the particular web page.

Anubhav-soin · 04-11-2016, 12:02 AM

Its a file that is used by search engines to find out that which pages in your website you want to crawled.

Note : do not edit the robot.txt yourself if you do not know how to modify it otherwise you good pages might get blocked from search engine.

You can also use site:www.yousite.com in google to find what are the pages that are currently been crawled from your website.

williamstinner · 04-11-2016, 10:51 PM

Robots.txt file is used to give instruction to the search engine crawler to which page has to crawled or not by allowing or disallowing.

jasonroy21 · 11-24-2016, 11:27 PM

The file tells the search engine crawlers which parts to web and which parts to leave alone in a website, differing between what is viewable to the public and what is viewable to the creators of the website alone.

03-05-2016, 01:39 AM	#1
Martinricky Registered User Join Date: Aug 2015 Posts: 434	What is robot.txt file and what are the benefits of using it? Hello, What is robot.txt file and what are the benefits of using it? Thank you __________________ Dentist in Delhi \| Best Dentist in Ashok Vihar \| Best Dentist in Shalimar Bagh \| Best Prosthodontist in Delhi

03-05-2016, 04:23 AM	#2
aditya12 Registered User Join Date: Jan 2016 Location: India Posts: 271	Robot.txt file is used to provide instruction to the search engine crawler about the webpages that which page to be crawled and which not. Best schools in Chandigarh Best schools in Mohali Best schools in Panchkula Best schools in Tricity Colleges in Chandigarh Colleges in Mohali Colleges in Tricity Colleges in Panchkula __________________ http://apnitricity.com/

03-05-2016, 04:52 AM	#3
tenso500 Registered User Join Date: Feb 2016 Posts: 51	The file tells the search engine crawlers which parts to web and which parts to leave alone in a website, differing between what is viewable to the public and what is viewable to the creators of the website alone. __________________ SocialSpinner - Smart Traffic Exchange

03-05-2016, 05:17 AM	#4
Admyrin Registered User Join Date: Feb 2016 Location: Surat Posts: 396	Robot.txt file is used to provide instruction to the search engine crawler about the webpages that which page to be crawled and which not. __________________ mother's day, mothers day quotes, mothers day images, mother day, mothers day gifts, mothers day gift ideas, mothers day cards, happy mothers day pictures mothers day messages mothers day quotes, mothers day pictures, Happy Mothers Day 2016, Happy Mothers Day SMS

03-24-2016, 06:46 AM	#5
langtu1292 Registered User Join Date: Mar 2016 Posts: 118	you can search on google..many thing for you __________________ iiStores Review

03-24-2016, 07:14 AM	#6
josechukkiri Registered User Join Date: Apr 2015 Location: India Posts: 34	The explanation about robot.txt files are satisfactory to me and thanks to the concerned. __________________ make money online Affiliates

03-25-2016, 05:57 AM	#7
sudeepkhana Registered User Join Date: Jan 2016 Location: India Posts: 1,258	Robot.txt is a text file kept on the web servers. The robot.txt allows the site admin to permit/allow which Search engines(like google, yahoo,bing) can crawl the site. Some times the admins won't allow any of the search engines to crawl at all. __________________ Packers and Movers in Cuttack \| packers and movers in dhenkanal

03-25-2016, 06:06 AM	#8
Devin Mataka Registered User Join Date: Mar 2016 Location: New York Posts: 2	The robots.txt file is a simple text file (no html) that is placed in your website’s root directory in order to tell the search engines which pages to index and which to skip. Many webmasters utilize this file to help the search engines index the content of their websites. If webmasters can tell the search engine spiders to skip pages that they do not consider important enough to be crawled (eg. printable versions of pages, .pdf files etc.), then they have a better opportunity to have their most valuable pages featured in the search engine results pages.The robots.txt file is a simple method of essentially easing the process for the spiders to return the most relevant search results. Another benefit of having a robots.txt is that you can specify the location of the Google .xml or Yahoo sitemap. This also increases spiderability for the search engines.

04-07-2016, 09:42 PM	#9
rahul3214 Registered User Join Date: Mar 2016 Posts: 75	Robots.txt file is important for seo, search engine spiders first see that where your robots.txt file because they do not know which file crawl or which not. In robots.txt file google know which file crawl. __________________ web development \| seo \| smo \| ppc \| digital marketing

04-08-2016, 12:27 PM	#10
karanjain Registered User Join Date: Jul 2015 Posts: 12	nice and awesome IPL 9 - Indian Premier League 2016 IPL Auction 2016 Live Results Shane Watson And Yuvraj Sold On Highest Bidding __________________ happy lohri Happy Lohri 2016 makar sankranti 2016

04-09-2016, 04:57 AM	#11
albertocosta Registered User Join Date: Apr 2016 Posts: 83	The robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website. __________________ Viaje a India 10 dias, agencias de Viajes India, Paquetes India, viajes india y nepal, Viajes en Grupo India, 1 Day - Jaipur City Tour, One Day Varanasi Tour

04-10-2016, 11:27 PM	#12
brknny Registered User Join Date: Feb 2016 Posts: 110	Robots.txt file is a file through which we give instructions to the search engines whether to crawl or not the particular web page. __________________ SEO online training IT training with live support PHP online training IT courses online training IT courses training

04-11-2016, 12:02 AM	#13
Anubhav-soin Registered User Join Date: Feb 2016 Location: Delhi , India Posts: 17	Its a file that is used by search engines to find out that which pages in your website you want to crawled. Note : do not edit the robot.txt yourself if you do not know how to modify it otherwise you good pages might get blocked from search engine. You can also use site:www.yousite.com in google to find what are the pages that are currently been crawled from your website. __________________ AmpCastOnline - Blogging For Everyone Last edited by Anubhav-soin; 04-11-2016 at 12:03 AM.. Reason: Improved answer

04-11-2016, 10:51 PM	#14
williamstinner Registered User Join Date: Mar 2016 Posts: 20	Robots.txt file is used to give instruction to the search engine crawler to which page has to crawled or not by allowing or disallowing.

11-24-2016, 11:27 PM	#15
jasonroy21 Registered User Join Date: Jul 2016 Posts: 755	The file tells the search engine crawlers which parts to web and which parts to leave alone in a website, differing between what is viewable to the public and what is viewable to the creators of the website alone. __________________ car cover \| car covers all weather car covers

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)