What is robots.txt? - Page 2 - Site Owners Forums

rolinsebra · 10-04-2016, 04:29 AM

Web site owners use the robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.

Ramkarma · 10-04-2016, 05:50 AM

Robot.txt :- Robot.txt is also known as the robots exclusion protocol(REP),is a text file webmaster create to instruct robots( typically search engine robots) how to crawl and index pages on their website. It is used to the new website when there is no content.

johnathan410 · 10-04-2016, 05:57 AM

Robots.txt is common name of a text file that is uploaded to a Web site's root directory and linked in the html code of the Web site. The robots.txt file is used to provide instructions about the Web site to Web robots and spiders.

ethanmichael201 · 10-04-2016, 11:16 PM

Robots.txt is a text file. it gives instruction to bots to crawlers about indexing and caching of a website or webpage.

nancy07 · 10-05-2016, 01:31 AM

The robots.txt file as instructions on where they are allowed to crawl (visit) and index (save) on the search engine results.*Robots.txt*files are useful: If you want search engines to ignore any duplicate pages on your website.

FerinKings · 10-05-2016, 03:12 AM

robots.txt is a text file, it indicates the crawler to which to crawl and which one don't want to crawl.

Jayasree · 10-05-2016, 03:22 AM

Bots will use robots.txt to crawl our website and webpages used for crawlers about indexing .

AshokDixit89 · 10-10-2016, 12:40 AM

The basic use of Robots.txt - The most common usage of Robots.txt is to ban crawlers from visiting private folders or content that gives them no additional information.

Robots.txt Allowing Access to Specific Crawlers.
Allow everything apart from certain patterns of URLs.

pattroderick · 10-11-2016, 01:48 AM

robots.txt is a file, that guides the crawler which one to crawl and which to not crawl..

autocarcovers · 10-11-2016, 01:56 AM

The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.

SaraSanjay · 10-12-2016, 10:49 PM

The robots avoidance convention (REP), or robots.txt is a content record website admins make to educate robots (normally web search tool robots) how to creep and file pages on their site.

NukeBlaster · 10-13-2016, 09:57 PM

Quote:

Originally Posted by abinaya

Robots.txt is a text file. It is through this file, it gives instruction to search engine crawlers about indexing and caching of a webpage, file of a website or directory, domain.

I first thought you wanted to ask until I realized you answered your own question.

jordanangel · 10-15-2016, 02:22 AM

The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.

saravjeet · 10-17-2016, 02:24 AM

The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.

jamesvincent21 · 10-17-2016, 02:31 AM

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.

10-04-2016, 04:29 AM	#16
rolinsebra Registered User Join Date: Oct 2016 Posts: 2	Web site owners use the robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. Last edited by rolinsebra; 11-12-2016 at 07:58 AM..

10-04-2016, 05:50 AM	#17
Ramkarma Registered User Join Date: Sep 2016 Posts: 63	Robot.txt :- Robot.txt is also known as the robots exclusion protocol(REP),is a text file webmaster create to instruct robots( typically search engine robots) how to crawl and index pages on their website. It is used to the new website when there is no content. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

10-04-2016, 05:57 AM	#18
johnathan410 Registered User Join Date: Aug 2016 Posts: 61	Robots.txt is common name of a text file that is uploaded to a Web site's root directory and linked in the html code of the Web site. The robots.txt file is used to provide instructions about the Web site to Web robots and spiders. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

10-04-2016, 11:16 PM	#19
ethanmichael201 Registered User Join Date: Jun 2016 Posts: 218	Robots.txt is a text file. it gives instruction to bots to crawlers about indexing and caching of a website or webpage. __________________ *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.*

10-05-2016, 01:31 AM	#20
nancy07 Registered User Join Date: Apr 2016 Posts: 217	The robots.txt file as instructions on where they are allowed to crawl (visit) and index (save) on the search engine results.Robots.txtfiles are useful: If you want search engines to ignore any duplicate pages on your website. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

10-05-2016, 03:12 AM	#21
FerinKings Registered User Join Date: Aug 2016 Posts: 205	robots.txt is a text file, it indicates the crawler to which to crawl and which one don't want to crawl. __________________ *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.*

10-05-2016, 03:22 AM	#22
Jayasree Registered User Join Date: Oct 2016 Posts: 252	Bots will use robots.txt to crawl our website and webpages used for crawlers about indexing .

10-10-2016, 12:40 AM	#23
AshokDixit89 Registered User Join Date: Jan 2016 Location: Mumbai, India Posts: 1,064	The basic use of Robots.txt - The most common usage of Robots.txt is to ban crawlers from visiting private folders or content that gives them no additional information. Robots.txt Allowing Access to Specific Crawlers. Allow everything apart from certain patterns of URLs. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

10-11-2016, 01:48 AM	#24
pattroderick Registered User Join Date: Jul 2016 Posts: 243	robots.txt is a file, that guides the crawler which one to crawl and which to not crawl.. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

10-11-2016, 01:56 AM	#25
autocarcovers Registered User Join Date: Sep 2016 Posts: 495	The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

10-12-2016, 10:49 PM	#26
SaraSanjay Registered User Join Date: May 2016 Posts: 551	The robots avoidance convention (REP), or robots.txt is a content record website admins make to educate robots (normally web search tool robots) how to creep and file pages on their site. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

10-15-2016, 02:22 AM	#28
jordanangel Registered User Join Date: Aug 2016 Posts: 387	The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.*

10-17-2016, 02:24 AM	#29
saravjeet Registered User Join Date: Oct 2016 Posts: 105	The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

10-17-2016, 02:31 AM	#30
jamesvincent21 Registered User Join Date: Sep 2016 Posts: 326	Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. __________________ *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.*

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Robots.txt	jaysh4922	Search Engine Optimization	13	07-28-2016 05:10 AM
Crawlers, spider & robots.txt?	jackthomas087	Search Engine Optimization	0	04-22-2014 03:29 AM
Evil bots consuming all my sites traffic even after i disallowed them in robots.txt	Fking	General Discussion	1	12-13-2013 11:27 PM
How to edit virtual robots.txt file in wordpress?	geniusoptimizer	PHP / mySQL	1	04-12-2013 12:36 PM
Unable to delete robots.txt file	notfake	Yahoo	0	04-29-2012 02:19 AM