What is robots.txt? - Page 3 - Site Owners Forums

Shiksha · 10-17-2016, 03:25 AM

It is great when search engines frequently visit your site and index your content but often there are cases when indexing parts of your online content is not what you want. For instance, if you have two versions of a page (one for viewing in the browser and one for printing), you'd rather have the printing version excluded from crawling, otherwise you risk being imposed a duplicate content penalty. Also, if you happen to have sensitive data on your site that you do not want the world to see, you will also prefer that search engines do not index these pages (although in this case the only sure way for not indexing sensitive data is to keep it offline on a separate machine). Additionally, if you want to save some bandwidth by excluding images, stylesheets and javascript from indexing, you also need a way to tell spiders to keep away from these items

Shiksha · 10-17-2016, 03:29 AM

The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned. Robots are often used by search engines to categorize web sites. Not all robots cooperate with the standard; email harvesters, spambots, malware, and robots that scan for security vulnerabilities may even start with the portions of the website where they have been told to stay out. The standard is different from, but can be used in conjunction with, Sitemaps, a robot inclusion standard for websites.

Jayasree · 02-13-2017, 04:09 AM

Robot.txt is a text file.It is used when the web page is new and there is no content in the web page, therefore it is important for SEO.

quocanh123 · 02-13-2017, 07:45 AM

The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.

jamesandy · 05-02-2017, 04:54 AM

Robots.txt is a file associated with your website used to ask different web crawlers to crawl or not crawl portions of your website.

AliceArifova · 05-02-2017, 10:36 PM

It is a type of text file which tells bot what to crawl or what to not ?

Hostingsafety · 05-02-2017, 10:41 PM

Robots.txt is common name of a text file that is uploaded to a Web site's root directory and linked in the html code of the Web site.

Jayasree · 05-02-2017, 11:06 PM

Robots.txt is a text file which will instruct search engine machine to crawl and index pages on your website.

komal14 · 05-03-2017, 03:26 AM

This file is used by search engine for crawling website's page and for given them index.

jumad · 05-03-2017, 04:59 AM

A robot.txt file instructs search bots about the pages that has to be indexed.

Seo in dubai

10-17-2016, 03:25 AM	#31
Shiksha Registered User Join Date: Oct 2016 Location: Chennai Posts: 11	It is great when search engines frequently visit your site and index your content but often there are cases when indexing parts of your online content is not what you want. For instance, if you have two versions of a page (one for viewing in the browser and one for printing), you'd rather have the printing version excluded from crawling, otherwise you risk being imposed a duplicate content penalty. Also, if you happen to have sensitive data on your site that you do not want the world to see, you will also prefer that search engines do not index these pages (although in this case the only sure way for not indexing sensitive data is to keep it offline on a separate machine). Additionally, if you want to save some bandwidth by excluding images, stylesheets and javascript from indexing, you also need a way to tell spiders to keep away from these items __________________ Education Management Software \| School Management Software \| Student Fees Management Software \| Library Management Software \| Shiksha Education Management Software \| Student Fees Collection Software \| SMS School Management Software \| SMS Software For Schools \| Students Performance Software \| School Result Management Software Last edited by Shiksha; 10-17-2016 at 03:31 AM..

10-17-2016, 03:29 AM	#32
Shiksha Registered User Join Date: Oct 2016 Location: Chennai Posts: 11	The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned. Robots are often used by search engines to categorize web sites. Not all robots cooperate with the standard; email harvesters, spambots, malware, and robots that scan for security vulnerabilities may even start with the portions of the website where they have been told to stay out. The standard is different from, but can be used in conjunction with, Sitemaps, a robot inclusion standard for websites. __________________ Education Management Software \| School Management Software \| Student Fees Management Software \| Library Management Software \| Shiksha Education Management Software \| Student Fees Collection Software \| SMS School Management Software \| SMS Software For Schools \| Students Performance Software \| School Result Management Software

02-13-2017, 04:09 AM	#33
Jayasree Registered User Join Date: Oct 2016 Posts: 252	Robot.txt is a text file.It is used when the web page is new and there is no content in the web page, therefore it is important for SEO. __________________ Point cook restaurant \| Melbourne Wedding venue \| Melbourne Resorts

02-13-2017, 07:45 AM	#34
quocanh123 Registered User Join Date: Feb 2017 Posts: 54	The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website. __________________ Rank hijack review Rank hijack bonus

05-02-2017, 04:54 AM	#35
jamesandy Registered User Join Date: Apr 2017 Posts: 13	Robots.txt is a file associated with your website used to ask different web crawlers to crawl or not crawl portions of your website. __________________ nissan frontier car covers nissan maxima car covers nissan murano car covers

05-02-2017, 10:36 PM	#36
AliceArifova Registered User Join Date: Sep 2016 Location: London Posts: 930	It is a type of text file which tells bot what to crawl or what to not ? __________________ Job Board wordpress themes , Job themes , Job Portal themes ,

05-02-2017, 10:41 PM	#37
Hostingsafety Registered User Join Date: Apr 2017 Location: Bangalore, White field Posts: 69	Robots.txt is common name of a text file that is uploaded to a Web site's root directory and linked in the html code of the Web site. __________________ Web Hosting India \| Domain Name Registration India \| cheap dedicated hosting India \| cheap dedicated server India

05-02-2017, 11:06 PM	#38
Jayasree Registered User Join Date: Oct 2016 Posts: 252	Robots.txt is a text file which will instruct search engine machine to crawl and index pages on your website. __________________ Point cook restaurant \| Melbourne Wedding venue \| Melbourne Resorts

05-03-2017, 03:26 AM	#39
komal14 Registered User Join Date: Apr 2017 Posts: 64	This file is used by search engine for crawling website's page and for given them index. __________________ Best school in Haryana \| Top school in Haryana \| Chartered Accountant in Delhi \| Boarding school in Haryana \| Top Interiors designing firms in Delhi NCR

05-03-2017, 04:59 AM	#40
jumad Registered User Join Date: Feb 2017 Posts: 10	A robot.txt file instructs search bots about the pages that has to be indexed. Seo in dubai

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Robots.txt	jaysh4922	Search Engine Optimization	13	07-28-2016 05:10 AM
Crawlers, spider & robots.txt?	jackthomas087	Search Engine Optimization	0	04-22-2014 03:29 AM
Evil bots consuming all my sites traffic even after i disallowed them in robots.txt	Fking	General Discussion	1	12-13-2013 11:27 PM
How to edit virtual robots.txt file in wordpress?	geniusoptimizer	PHP / mySQL	1	04-12-2013 12:36 PM
Unable to delete robots.txt file	notfake	Yahoo	0	04-29-2012 02:19 AM