What is the use of robots.txt? - Site Owners Forums

bellevoir · 09-07-2016, 04:46 AM

What is the use of robots.txt?

webdesigndubai · 09-07-2016, 09:32 AM

Robot.txt file is use to disallow pages which having confidential information

alishasmith · 09-07-2016, 11:33 PM

Robots.txt is a text file which gives the instructions to crawlers about the website that which pages should be crawled or not.

Pillars · 09-08-2016, 02:52 AM

Robots.txt is a text file that is inserted into your website and contains information for search engine robots. The file lists webpages that are allowed and disallowed from search engine crawling.

illuminatingasi · 09-08-2016, 02:57 AM

Robots.txt file allows to search engine spider which one page of website will be cache.

RH-Calvin · 09-12-2016, 11:05 PM

Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.

cedric903 · 09-13-2016, 02:31 AM

Thanks, it very useful.

George re · 09-13-2016, 04:32 AM

Robots.txt is a text file which gives the instructions to crawlers about the website that which pages should be crawled or not.

jasonroy21 · 09-14-2016, 03:14 AM

Robots.txt is a text file which gives the instructions to crawlers about the website that will not allow to visit

AshokDixit89 · 09-19-2016, 12:59 AM

Use of Robots.txt - The most common usage of Robots.txt is to ban crawlers from visiting private folders or content that gives them no additional information.

Robots.txt Allowing Access to Specific Crawlers.
Allow everything apart from certain patterns of URLs.

lisaryan · 09-19-2016, 05:11 AM

The robots.txt file helps search engines understand which pages on a website should be indexed and crawled. This file can also be used to block the unwanted crawlers to search your websites.

pxljobs · 07-06-2017, 11:49 PM

These file give permission to search engine crawler which one to access and which one to not access these are helpful to improve the security.

alexson · 07-07-2017, 03:01 AM

Through robots.txt you can stop search bots to crawl your websites. You can also use meta robots to prevent your website or web page.

newproaudio · 07-07-2017, 03:16 AM

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.

manavatmix · 07-07-2017, 05:03 PM

The robots file tells search engine bots which pages to index & which ones to ignore.

09-07-2016, 04:46 AM	#1
bellevoir Registered User Join Date: Jan 2016 Location: 3579 E. Foothill Blvd. #190 Pasadena, CA 91107-3119 Posts: 66	What is the use of robots.txt? What is the use of robots.txt? __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

09-07-2016, 09:32 AM	#2
webdesigndubai Registered User Join Date: Sep 2016 Location: Dubai, UAE Posts: 1	Robot.txt file is use to disallow pages which having confidential information __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

09-07-2016, 11:33 PM	#3
alishasmith Registered User Join Date: Aug 2016 Location: Ghaziabad Posts: 46	Robots.txt is a text file which gives the instructions to crawlers about the website that which pages should be crawled or not. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.*

09-08-2016, 02:52 AM	#4
Pillars Registered User Join Date: Aug 2015 Location: Chennai, TN, India Posts: 362	Robots.txt is a text file that is inserted into your website and contains information for search engine robots. The file lists webpages that are allowed and disallowed from search engine crawling. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

09-08-2016, 02:57 AM	#5
illuminatingasi Registered User Join Date: Aug 2015 Posts: 90	Robots.txt file allows to search engine spider which one page of website will be cache. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

09-12-2016, 11:05 PM	#6
RH-Calvin Registered User Join Date: May 2015 Location: Forum Posts: 2,517	Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling. __________________ █ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| $1 VPS Hosting █ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| Windows Server with Remote Desktop Access █ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| Servers with Free IPMI Setup

09-13-2016, 02:31 AM	#7
cedric903 Registered User Join Date: Jul 2016 Posts: 33	Thanks, it very useful.

09-13-2016, 04:32 AM	#8
George re Registered User Join Date: Sep 2016 Posts: 33	Robots.txt is a text file which gives the instructions to crawlers about the website that which pages should be crawled or not.

09-14-2016, 03:14 AM	#9
jasonroy21 Registered User Join Date: Jul 2016 Posts: 755	Robots.txt is a text file which gives the instructions to crawlers about the website that will not allow to visit __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

09-19-2016, 12:59 AM	#10
AshokDixit89 Registered User Join Date: Jan 2016 Location: Mumbai, India Posts: 1,064	Use of Robots.txt - The most common usage of Robots.txt is to ban crawlers from visiting private folders or content that gives them no additional information. Robots.txt Allowing Access to Specific Crawlers. Allow everything apart from certain patterns of URLs. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

09-19-2016, 05:11 AM	#11
lisaryan Registered User Join Date: Sep 2016 Location: Bangalore Posts: 58	The robots.txt file helps search engines understand which pages on a website should be indexed and crawled. This file can also be used to block the unwanted crawlers to search your websites.

07-06-2017, 11:49 PM	#12
pxljobs Registered User Join Date: Jul 2016 Location: Bangalore Posts: 370	These file give permission to search engine crawler which one to access and which one to not access these are helpful to improve the security. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

07-07-2017, 03:01 AM	#13
alexson Registered User Join Date: Jan 2017 Location: Arizona Posts: 153	Through robots.txt you can stop search bots to crawl your websites. You can also use meta robots to prevent your website or web page. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

07-07-2017, 03:16 AM	#14
newproaudio Registered User Join Date: May 2017 Posts: 269	Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

07-07-2017, 05:03 PM	#15
manavatmix Registered User Join Date: Jun 2016 Location: Delhi Posts: 436	The robots file tells search engine bots which pages to index & which ones to ignore. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Robots.txt	jaysh4922	Search Engine Optimization	13	07-28-2016 05:10 AM
Crawlers, spider & robots.txt?	jackthomas087	Search Engine Optimization	0	04-22-2014 03:29 AM
Evil bots consuming all my sites traffic even after i disallowed them in robots.txt	Fking	General Discussion	1	12-13-2013 11:27 PM
How to edit virtual robots.txt file in wordpress?	geniusoptimizer	PHP / mySQL	1	04-12-2013 12:36 PM
Unable to delete robots.txt file	notfake	Yahoo	0	04-29-2012 02:19 AM