What is robots.txt? - Site Owners Forums

eagleeye dental · 09-10-2014, 09:35 PM

What is robots.txt?

techgroupz · 09-10-2014, 10:04 PM

robot.txt is the text file its help to stop the crawler to index the page which is not allow in the robot.txt and its upload with help of the google webmaster tools and code of robot.txt look like such as the
user agent: *
Disallow /abc/
disallow /xyz/

stevenj29 · 09-10-2014, 10:58 PM

The particular robots exclusion protocol (REP) or robots(dot) txt is usually a textual content record website owners build to instruct robots (typically Google search robots ) they way to get in addition to directory web pages on their website.

mandypackage · 09-10-2014, 11:20 PM

yes agree with the both that robot.txt file is used to denied the google crawler to index some some pages of the pages or guide the crawler that which has to index and which not

rajivdevkk · 09-10-2014, 11:49 PM

Robots.txt file is really useful to disallow a particular page of the site.

disallow / abc

picnic spot in delhi | camp wild dhauj

Kalpart · 09-11-2014, 12:32 AM

Robot.txt is a text file which inform the crawlers or spiders weather they should visit the page or not.

ArthurHoagland · 09-11-2014, 01:17 AM

Robot.txt is used to give knowledge about any website to google. Search engines can know which thing has to be crawled in any website and which should not to be crawled with the help of robot.txt.

ubitestechnolog · 09-11-2014, 11:43 PM

Robots.txt is a file in the root directory of your web site that instructs web crawlers what parts, or all, or none of your site they are allowed examine.

sachinkar09 · 09-12-2014, 02:31 AM

User-agent: *
Disallow: /

it is used to prevent security breaking..

Akshay · 09-12-2014, 03:39 AM

Robots is the most useful file, you can control google crawling process by using robots.txt

09-10-2014, 09:35 PM	#1
eagleeye dental Registered User Join Date: Aug 2014 Location: India Posts: 88	What is robots.txt? What is robots.txt?

09-10-2014, 10:04 PM	#2
techgroupz Registered User Join Date: Aug 2014 Posts: 7	robot.txt is the text file its help to stop the crawler to index the page which is not allow in the robot.txt and its upload with help of the google webmaster tools and code of robot.txt look like such as the user agent: * Disallow /abc/ disallow /xyz/ __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

09-10-2014, 10:58 PM	#3
stevenj29 Registered User Join Date: Apr 2014 Posts: 16	The particular robots exclusion protocol (REP) or robots(dot) txt is usually a textual content record website owners build to instruct robots (typically Google search robots ) they way to get in addition to directory web pages on their website. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

09-10-2014, 11:20 PM	#4
mandypackage Registered User Join Date: Aug 2014 Posts: 8	yes agree with the both that robot.txt file is used to denied the google crawler to index some some pages of the pages or guide the crawler that which has to index and which not __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

09-11-2014, 12:32 AM	#6
Kalpart Registered User Join Date: Aug 2014 Posts: 344	Robot.txt is a text file which inform the crawlers or spiders weather they should visit the page or not. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

09-10-2014, 11:49 PM	#5
rajivdevkk Registered User Join Date: Sep 2013 Location: Delhi Posts: 344	Robots.txt file is really useful to disallow a particular page of the site. disallow / abc picnic spot in delhi \| camp wild dhauj

09-11-2014, 01:17 AM	#7
ArthurHoagland Registered User Join Date: Sep 2014 Posts: 25	Robot.txt is used to give knowledge about any website to google. Search engines can know which thing has to be crawled in any website and which should not to be crawled with the help of robot.txt. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

09-11-2014, 11:43 PM	#8
ubitestechnolog Registered User Join Date: Sep 2014 Posts: 14	Robots.txt is a file in the root directory of your web site that instructs web crawlers what parts, or all, or none of your site they are allowed examine.

09-12-2014, 02:31 AM	#9
sachinkar09 Registered User Join Date: Sep 2014 Posts: 2	User-agent: * Disallow: / it is used to prevent security breaking.. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

09-12-2014, 03:39 AM	#10
Akshay Registered User Join Date: Sep 2014 Posts: 10	Robots is the most useful file, you can control google crawling process by using robots.txt __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Crawlers, spider & robots.txt?	jackthomas087	Search Engine Optimization	0	04-22-2014 03:29 AM
Evil bots consuming all my sites traffic even after i disallowed them in robots.txt	Fking	General Discussion	1	12-13-2013 11:27 PM
How to edit virtual robots.txt file in wordpress?	geniusoptimizer	PHP / mySQL	1	04-12-2013 12:36 PM
The robots.txt file	rubberfender	Yahoo	0	12-02-2012 07:32 PM
Unable to delete robots.txt file	notfake	Yahoo	0	04-29-2012 02:19 AM