What is robots.txt? - Page 2 - Site Owners Forums

SARAHANSEN · 01-04-2017, 02:52 AM

The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.

santoshkumar · 01-04-2017, 02:58 AM

Thank you so much...

Pujitaharsha · 01-04-2017, 03:09 AM

The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.

ITSGOLIVE · 01-04-2017, 03:58 AM

Robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.

ethanmichael201 · 01-04-2017, 05:34 AM

Robots.txt, is a text file present in the root directory of a website. The Robots.txt file is a convention created to direct the activity of search engine crawlers or web spiders.

tionnasmith · 03-24-2017, 03:32 AM

Robots.txt is used to block the cawler from crawling the private areas.

justseeservice · 03-26-2017, 11:20 PM

Robots.txt is provide instructions to the crawlers about the caching and indexing of a website.

sashwatmegh · 03-26-2017, 11:31 PM

Robot.txt is a text file that specify how to inform the web robot about which areas of the website should not be processed or scanned.

jamesrobertt · 03-27-2017, 01:28 AM

The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.

jordanangel · 03-27-2017, 10:10 PM

Robots.txt is a standard used by websites to communicate with web crawlers and other web robots.

collaberex · 03-28-2017, 02:35 AM

The robots exclusion protocol is a standard used by the websites to communicate with web crawlers and other web robots

Sgraphinfotech · 03-28-2017, 02:56 AM

Robots.txt is one of the text fil, Which is used to instruct robots, how to crawl and index pages on their website.

sinGN · 03-28-2017, 03:30 AM

Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit.

justfeelus · 03-28-2017, 09:14 PM

an text file that indicates pages not to be crawled

sharmaroshni012 · 03-28-2017, 10:55 PM

A robots.txt file is a file at the root of your site that indicates those parts of your site you don’t want to be accessed by search engine crawlers. The file uses the Robots Exclusion Standard, which is a protocol with a small set of commands that can be used to indicate access to your site by section and by specific kinds of web crawlers (such as mobile crawlers vs desktop crawlers).

01-04-2017, 02:52 AM	#16
SARAHANSEN Registered User Join Date: Jan 2017 Posts: 9	The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned. __________________ ABS 415\|ABS 417\|UOP COURSES

01-04-2017, 02:58 AM	#17
santoshkumar Registered User Join Date: Oct 2016 Posts: 30	Thank you so much... __________________ Udaipur \| Kodaikanal \| Shimla \| Manali \| Nainital \| Mussoorie \| Dalhousie \| Dharamshala

01-04-2017, 03:09 AM	#18
Pujitaharsha Registered User Join Date: Nov 2016 Posts: 72	The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned. __________________ Business Listing Software Company List Outsource to India

01-04-2017, 03:58 AM	#19
ITSGOLIVE Registered User Join Date: Dec 2016 Posts: 112	Robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned. __________________ sms service branded sms in pakistan sms short code Bulk sms services \| sms marketing

01-04-2017, 05:34 AM	#20
ethanmichael201 Registered User Join Date: Jun 2016 Posts: 218	Robots.txt, is a text file present in the root directory of a website. The Robots.txt file is a convention created to direct the activity of search engine crawlers or web spiders. __________________ Best SEO company USA \| SEO services USA \| Best online branding company USA

03-24-2017, 03:32 AM	#21
tionnasmith Registered User Join Date: Sep 2016 Posts: 226	Robots.txt is used to block the cawler from crawling the private areas. __________________ Volkswagen Eos car covers \| Porsche Panamera car covers

03-26-2017, 11:20 PM	#22
justseeservice Registered User Join Date: Mar 2017 Posts: 25	Robots.txt is provide instructions to the crawlers about the caching and indexing of a website. __________________ Thyroid Clinic in Chennai \| Endocrine Doctor in Chennai \| Endocrinology Clinic in Chennai

03-26-2017, 11:31 PM	#23
sashwatmegh Registered User Join Date: Apr 2016 Location: pune Posts: 583	Robot.txt is a text file that specify how to inform the web robot about which areas of the website should not be processed or scanned. __________________ Accounting Software in Chhattisgarh\|HRM software in Chhattisgarh

03-27-2017, 01:28 AM	#24
jamesrobertt Registered User Join Date: Dec 2016 Posts: 530	The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned. __________________ Preach the Gospel Help the homeless How To Be Saved Global Ministry Baptism Five Fold Ministry Online Minister http://servinglove.ca/

03-27-2017, 10:10 PM	#25
jordanangel Registered User Join Date: Aug 2016 Posts: 387	Robots.txt is a standard used by websites to communicate with web crawlers and other web robots. __________________ Pop up tents Event tents Wedding Tents

03-28-2017, 02:35 AM	#26
collaberex Registered User Join Date: Oct 2016 Posts: 335	The robots exclusion protocol is a standard used by the websites to communicate with web crawlers and other web robots __________________ lead generation face to face networking networking for accountants

03-28-2017, 02:56 AM	#27
Sgraphinfotech Registered User Join Date: Mar 2017 Location: Bangalore Posts: 14	Robots.txt is one of the text fil, Which is used to instruct robots, how to crawl and index pages on their website. __________________ oracle sql training in bangalore \| oracle sql training institutes in bangalore \| oracle sql training center in bangalore \| oracle sql training course in bangalore \| sql course training in bangalore

03-28-2017, 03:30 AM	#28
sinGN Registered User Join Date: Nov 2016 Posts: 330	Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit. __________________ Custom Hoodies, Customized Tshirts, Custom Sport Jersey, Mug Printing, Photo Printed Mugs, Corporate tshirts

03-28-2017, 09:14 PM	#29
justfeelus Registered User Join Date: Mar 2017 Location: chennai Posts: 71	an text file that indicates pages not to be crawled

03-28-2017, 10:55 PM	#30
sharmaroshni012 Registered User Join Date: Nov 2016 Posts: 412	A robots.txt file is a file at the root of your site that indicates those parts of your site you don’t want to be accessed by search engine crawlers. The file uses the Robots Exclusion Standard, which is a protocol with a small set of commands that can be used to indicate access to your site by section and by specific kinds of web crawlers (such as mobile crawlers vs desktop crawlers). __________________ How to start online business in India \| How to start ecommerce business in India

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

Thread Tools
Show Printable Version Email this Page
Display Modes	Rate This Thread
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode	Rate This Thread:

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Robots.txt	jaysh4922	Search Engine Optimization	13	07-28-2016 05:10 AM
Crawlers, spider & robots.txt?	jackthomas087	Search Engine Optimization	0	04-22-2014 03:29 AM
Evil bots consuming all my sites traffic even after i disallowed them in robots.txt	Fking	General Discussion	1	12-13-2013 11:27 PM
How to edit virtual robots.txt file in wordpress?	geniusoptimizer	PHP / mySQL	1	04-12-2013 12:36 PM
Unable to delete robots.txt file	notfake	Yahoo	0	04-29-2012 02:19 AM