What is robots.txt used for? - Page 2 - Site Owners Forums

jackparrow · 11-17-2016, 09:53 PM

robots.txt file is used for control search engine bot.

Amirthi · 11-17-2016, 10:22 PM

Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit.

LukeBarter · 11-17-2016, 10:23 PM

robots.txt is a text file webmasters create to instruct robots how to crawl and index pages on their website.

abhishek1 · 11-21-2016, 12:00 AM

The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.

AshokDixit89 · 11-21-2016, 12:40 AM

Use of Robots.txt - The most common usage of Robots.txt is to ban crawlers from visiting private folders or content that gives them no additional information.

Robots.txt Allowing Access to Specific Crawlers.
Allow everything apart from certain patterns of URLs.

Dentcare · 11-21-2016, 05:21 AM

Robots.txt file give instruction to search engine spiders, how search engine spider to see and interact with your web pages.

nathanleo · 01-28-2022, 03:41 AM

A robots. txt file tells search engine crawlers which URLs the crawler can access on your site. This is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google.

11-17-2016, 09:53 PM	#16
jackparrow Registered User Join Date: Oct 2016 Posts: 128	robots.txt file is used for control search engine bot. __________________ patio umbrellas \| patio sun umbrellas \| green cantilever umbrella

11-17-2016, 10:22 PM	#17
Amirthi Registered User Join Date: Nov 2016 Posts: 46	Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit. __________________ Terrace garden in Chennai garden service Chennai\|landscaping Chennai \|garden maintenance in Chennai\|landscaping contractors Chennai\|gardening contractors Chennai\|landscape designer Chennai\|landscape services Chennai

11-17-2016, 10:23 PM	#18
LukeBarter Registered User Join Date: Sep 2016 Posts: 171	robots.txt is a text file webmasters create to instruct robots how to crawl and index pages on their website. __________________ *Exchange Second Hand Goods* \|\| *Free Classifieds in India*

11-21-2016, 12:00 AM	#19
abhishek1 Registered User Join Date: Oct 2016 Posts: 71	The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website. __________________ Water softener dealer Grundfos pump supplier Pressure Pump Service

11-21-2016, 12:40 AM	#20
AshokDixit89 Registered User Join Date: Jan 2016 Location: Mumbai, India Posts: 1,064	Use of Robots.txt - The most common usage of Robots.txt is to ban crawlers from visiting private folders or content that gives them no additional information. Robots.txt Allowing Access to Specific Crawlers. Allow everything apart from certain patterns of URLs. __________________ Screener\|Equity Research

11-21-2016, 05:21 AM	#21
Dentcare Registered User Join Date: Aug 2016 Posts: 187	Robots.txt file give instruction to search engine spiders, how search engine spider to see and interact with your web pages. __________________ Best Braces Treatment In Chennai \| Dental Clinic In Besant Nagar \| Best Dental Clinic In Besant \| Dental Clinic In Thiruvanmiyur \| Best Dental Clinic In Thiruvanmiyur \| Dental Clinic In Mylapore \| Best Dental Clinic In Mylapore

01-28-2022, 03:41 AM	#22
nathanleo Registered User Join Date: Jul 2017 Posts: 2,095	A robots. txt file tells search engine crawlers which URLs the crawler can access on your site. This is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google. __________________ Residential Plots for Sale in ECR Plots for Sale in ECR Villa Plots in siruseri Plots Sale in Kelambakkam Near By Plot for sale Residential Plots in Kelambakkam Plot for sale in OMR

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Crawlers, spider & robots.txt?	jackthomas087	Search Engine Optimization	0	04-22-2014 03:29 AM
Evil bots consuming all my sites traffic even after i disallowed them in robots.txt	Fking	General Discussion	1	12-13-2013 11:27 PM
How to edit virtual robots.txt file in wordpress?	geniusoptimizer	PHP / mySQL	1	04-12-2013 12:36 PM
The robots.txt file	rubberfender	Yahoo	0	12-02-2012 07:32 PM
Unable to delete robots.txt file	notfake	Yahoo	0	04-29-2012 02:19 AM