What is robots.txt? - Site Owners Forums

praveen007 · 05-22-2014, 02:56 AM

It is a kind of text file utilized to provide the instructions to the crawlers about the caching and indexing of a website, directory, domain or a file of a webpage.

larrylo · 05-22-2014, 03:15 AM

You have covered up the basic definition of what it means

austin11 · 05-22-2014, 03:59 AM

Robots.txt helps to index ourwebsite all pages on google.

AvniShergill · 05-23-2014, 12:55 AM

Robots.txt files are used to inform the crawler which URL to crawl and which to avoid.

rummy26 · 05-23-2014, 02:08 AM

Robots.txt files are used to inform the search engine crawler to which URL to allow search engines and which is to disallow .

iedigcom · 05-23-2014, 03:30 AM

Crawling of website by robots is first step to getting ranked in SearchEngines. The Robot Exclusion Standard, also known as the Robots Exclusion Protocol or robots.txt protocol. Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.

DanielGarneau · 05-23-2014, 03:36 AM

By way of example, my website is css and html based, which means that it is not currently built with a CMS such as WordPress' and I use the robot.txt file to excule search engines from reaching the development directories I use to test drive CMS softwares such as WordPress. When my WordPress based site is ready for being seen by the world, I will remove the exclusion line from my robot.txt.

You can find the syntax supported by robot.txt files by doing a google search. They will also tell it where to put it on your site, which is the same directory where your index.html file is. If you use a CMS like WordPress, I believe the software generates the robot.txt file according to the parameters you choose within WordPress, but I'm less familiar with these than with the css and html based web sites.

HarveyParker · 05-23-2014, 03:39 AM

Robot.txt is a text file which you put in your website to tell search engine which pages you would like not to crawl.
For an Example:

User Agest: *

Disallow:/admin/
Disallow:/epe/de
Disallow:/dwsdf/fdedx/

ChristinaCa · 05-24-2014, 01:09 AM

Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit.

sarvodyaseo · 05-24-2014, 01:17 AM

Robots.txt is a text file for gives the intruction of bots. particular search engine bots or all search engine bots which file or folder have cached or not cached.

yugant · 10-13-2014, 06:38 AM

robots.txt is a text file that search engine information allow or not.

Rajumakkar07 · 10-13-2014, 01:33 PM

Its pretty much a file that search engines look at to figure out whether to index your site or not.

ezzholidays · 10-13-2014, 11:31 PM

it is a .txt file, that doesn't crawled by the web spider.

ledtubelight · 10-14-2014, 12:05 AM

thanks for the information........

bestindiatours · 10-14-2014, 12:12 AM

its the text file which with the name robot.txt use to disallow the crawler to crawl the page that we don't need to crawl

05-22-2014, 02:56 AM	#1
praveen007 Registered User Join Date: Mar 2014 Posts: 96	What is robots.txt? It is a kind of text file utilized to provide the instructions to the crawlers about the caching and indexing of a website, directory, domain or a file of a webpage. __________________ Bluetooth Earpiece ! Secret Software

05-22-2014, 03:15 AM	#2
larrylo Registered User Join Date: May 2014 Posts: 9	You have covered up the basic definition of what it means __________________ affiliate marketing programs

05-22-2014, 03:59 AM	#3
austin11 Registered User Join Date: Oct 2013 Posts: 77	Robots.txt helps to index ourwebsite all pages on google. __________________ Rehabilitation centre \| Sukhbir Singh Badal \| Sukhbir Badal

05-23-2014, 12:55 AM	#4
AvniShergill Registered User Join Date: May 2014 Posts: 89	Robots.txt files are used to inform the crawler which URL to crawl and which to avoid. __________________ Blasting Machine \| Almen Strips

05-23-2014, 02:08 AM	#5
rummy26 Registered User Join Date: Nov 2010 Location: us Posts: 584	Robots.txt files are used to inform the search engine crawler to which URL to allow search engines and which is to disallow . __________________ Mobile App Development Company Calgary App Marketing Property in Clerkenwell

05-23-2014, 03:30 AM	#6
iedigcom Registered User Join Date: May 2014 Posts: 16	Crawling of website by robots is first step to getting ranked in SearchEngines. The Robot Exclusion Standard, also known as the Robots Exclusion Protocol or robots.txt protocol. Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. __________________ Bhojpuri News \| Fundo Job \| Download MP3 \| Bhojpuri Film News

05-23-2014, 03:36 AM	#7
DanielGarneau Registered User Join Date: May 2014 Location: Quebec City, Quebec, Canada Posts: 23	By way of example, my website is css and html based, which means that it is not currently built with a CMS such as WordPress' and I use the robot.txt file to excule search engines from reaching the development directories I use to test drive CMS softwares such as WordPress. When my WordPress based site is ready for being seen by the world, I will remove the exclusion line from my robot.txt. You can find the syntax supported by robot.txt files by doing a google search. They will also tell it where to put it on your site, which is the same directory where your index.html file is. If you use a CMS like WordPress, I believe the software generates the robot.txt file according to the parameters you choose within WordPress, but I'm less familiar with these than with the css and html based web sites. __________________ Daniel Garneau www.savoiretcroire.ca -- Knowing and Believing: Torn-apart, yet inseparable allies… Last edited by DanielGarneau; 05-23-2014 at 03:41 AM..

05-23-2014, 03:39 AM	#8
HarveyParker Registered User Join Date: May 2014 Location: Mumbai Posts: 6	Robot.txt is a text file which you put in your website to tell search engine which pages you would like not to crawl. For an Example: User Agest: * Disallow:/admin/ Disallow:/epe/de Disallow:/dwsdf/fdedx/ __________________ Party Planet is one of India’s best Party Planner like Kids Birthday Party Celebration, Themes For Kids Birthday Party & Corporate Events in Mumbai Location.

05-24-2014, 01:09 AM	#9
ChristinaCa Registered User Join Date: Mar 2014 Posts: 210	Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit. __________________ Kids Birthday Party \| Kids Birthday Parties

05-24-2014, 01:17 AM	#10
sarvodyaseo Registered User Join Date: Mar 2014 Posts: 86	Robots.txt is a text file for gives the intruction of bots. particular search engine bots or all search engine bots which file or folder have cached or not cached. __________________ Custom Website Designing Company \| Mobile Apps Development Company \| Software Development Company \| Website Design Company \| Yearly Horoscope\| Latest Fashion Trends in India for Womens \| Buy Online Designer Clothes for men \| House Party Planners in Delhi NCR

10-13-2014, 06:38 AM	#11
yugant Registered User Join Date: Sep 2014 Posts: 228	robots.txt is a text file that search engine information allow or not. __________________ http://fireip.com

10-13-2014, 01:33 PM	#12
Rajumakkar07 Registered User Join Date: Oct 2014 Location: Mississauga Posts: 11	Its pretty much a file that search engines look at to figure out whether to index your site or not. __________________ Knowing is not enough, we must apply. Willing is not enough, we must do. Toronto Web Design \| Mississauga Web Design

10-13-2014, 11:31 PM	#13
ezzholidays Registered User Join Date: Sep 2014 Location: Bangalore Posts: 20	it is a .txt file, that doesn't crawled by the web spider. __________________ Tour Packages from Chennai \| Honeymoon Packages from chennai \| Tour Packages from Bangalore

10-14-2014, 12:05 AM	#14
ledtubelight Registered User Join Date: Aug 2014 Location: Surat Posts: 847	thanks for the information........ __________________ I am SEO expert and handling SEO for following words led tube light India, Send cakes Surat, Send Flowers cakes Surat, Led Tube Light Surat send flowers to surat Buy cakes in Surat Buy flowers in surat Send ONLINE CAKES FLOWERS SURAT

10-14-2014, 12:12 AM	#15
bestindiatours Registered User Join Date: Oct 2014 Posts: 5	its the text file which with the name robot.txt use to disallow the crawler to crawl the page that we don't need to crawl

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Crawlers, spider & robots.txt?	jackthomas087	Search Engine Optimization	0	04-22-2014 03:29 AM
Evil bots consuming all my sites traffic even after i disallowed them in robots.txt	Fking	General Discussion	1	12-13-2013 11:27 PM
How to edit virtual robots.txt file in wordpress?	geniusoptimizer	PHP / mySQL	1	04-12-2013 12:36 PM
The robots.txt file	rubberfender	Yahoo	0	12-02-2012 07:32 PM
Unable to delete robots.txt file	notfake	Yahoo	0	04-29-2012 02:19 AM