how to use robot.txt? - Site Owners Forums

dean1122 · 12-05-2012, 08:07 AM

hi,
What is robot.txt in SEO and how to install .
Please friends share your view.
Thanks

quicklockworks · 12-05-2012, 08:48 AM

robots.txt is a set of instructions to let the search engines know where and what to look at on your site.

you out it in the top-level directory of your web server.

read a couple articles before doing it though because there is much more detail then this

RussF · 12-05-2012, 02:26 PM

If you don't have a robot.txt the search engines just crawl every page of your website. There's a lot of articles on how to use the file.

laceywilliams12 · 12-05-2012, 10:12 PM

Robot.txt is a text file which have set of instructions for crawlers ot crawl the website or a specific page. If you have it then it must be placed in root directory.....and for more help you can read lots Articles related to this...thanks

arindamdutta16 · 12-06-2012, 03:51 AM

The robots.txt is a simple text file in your web site that informs search engine bots how to crawl and index website or web pages. It is great when search engines frequently visit your site and index your content but often there are cases when indexing parts of your online content is not what you want.

john mathew · 12-29-2012, 04:53 AM

Robots.txt is a file that is used to exclude content from the crawling process of search engine spiders / bots. Robots.txt is also called the Robots Exclusion Protocol.

jaysh4922 · 12-31-2012, 02:15 AM

type this in
User-agent: * Disallow:

this is an example... open notepad, and just save a robots, then upload to your root.

kevinloyed · 01-01-2013, 12:05 AM

Robots.txt is a file that is used to exclude content from the crawling process of search engine spiders / bots. Robots.txt.

rummy26 · 01-01-2013, 11:27 PM

Quote:

Originally Posted by jaysh4922

type this in
User-agent: * Disallow:

this is an example... open notepad, and just save a robots, then upload to your root.

Only this code is enough?
By adding this code which pages search engines will not visit?

likit · 01-02-2013, 06:44 AM

robots.txt is a set of instructions to tell the search engines where and what to look at on your site.

laceywilliams12 · 01-03-2013, 04:49 AM

robots.txt is a set of instructions to let the search engines know where and what to look for on your site.

sher.winchan58 · 01-05-2013, 02:10 AM

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.

It works likes this: a robot wants to vists a Web site URL, say http://www.example.com/welcome.html. Before it does so, it firsts checks for http://www.example.com/robots.txt, and finds:
User-agent: *
Disallow: /

The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

There are two important considerations when using /robots.txt:

robots can ignore your /robots.txt. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention.
the /robots.txt file is a publicly available file. Anyone can see what sections of your server you don't want robots to use.

12-05-2012, 08:07 AM	#1
dean1122 Registered User Join Date: Sep 2012 Posts: 39	how to use robot.txt? hi, What is robot.txt in SEO and how to install . Please friends share your view. Thanks __________________ comprehensive internet marketing and seo service www.cimass.com

12-05-2012, 08:48 AM	#2
quicklockworks Registered User Join Date: Aug 2012 Posts: 120	robots.txt is a set of instructions to let the search engines know where and what to look at on your site. you out it in the top-level directory of your web server. read a couple articles before doing it though because there is much more detail then this __________________ Locksmith Calgary Calgary Locksmith

12-05-2012, 02:26 PM	#3
RussF Registered User Join Date: Dec 2012 Posts: 25	If you don't have a robot.txt the search engines just crawl every page of your website. There's a lot of articles on how to use the file. __________________ ? ? XenLayer - Paravirtualization Professionals since 2008 - [ICQ ME] ? ? Reseller Hosting \| OpenVZ VPS \| XEN VPS \| Dedicated Servers

12-06-2012, 03:51 AM	#5
arindamdutta16 Registered User Join Date: Nov 2012 Location: India Posts: 103	The robots.txt is a simple text file in your web site that informs search engine bots how to crawl and index website or web pages. It is great when search engines frequently visit your site and index your content but often there are cases when indexing parts of your online content is not what you want. __________________ Best Web Design Services - Corporate Web Design, Creative Design, Custom Web Design, Website Re-Design Quality SEO Services - - SEO, Pay Per Click (PPC), SEM, Link Popularity Building, SEO Copywriting, SMO, SMM

12-29-2012, 04:53 AM	#6
john mathew Registered User Join Date: Feb 2012 Posts: 225	Robots.txt is a file that is used to exclude content from the crawling process of search engine spiders / bots. Robots.txt is also called the Robots Exclusion Protocol. __________________ fixed term loan \| apply for a home loan

12-05-2012, 10:12 PM	#4
laceywilliams12 Registered User Join Date: Nov 2012 Posts: 71	Robot.txt is a text file which have set of instructions for crawlers ot crawl the website or a specific page. If you have it then it must be placed in root directory.....and for more help you can read lots Articles related to this...thanks

12-31-2012, 02:15 AM	#7
jaysh4922 Registered User Join Date: Dec 2012 Posts: 657	type this in User-agent: * Disallow: this is an example... open notepad, and just save a robots, then upload to your root. __________________ Web Development Services SEO Services

01-01-2013, 12:05 AM	#8
kevinloyed Registered User Join Date: Mar 2012 Location: india Posts: 218	Robots.txt is a file that is used to exclude content from the crawling process of search engine spiders / bots. Robots.txt. __________________ Web development Mobile application development iPhone application development

01-02-2013, 06:44 AM	#10
likit Registered User Join Date: Oct 2012 Posts: 149	robots.txt is a set of instructions to tell the search engines where and what to look at on your site.

01-03-2013, 04:49 AM	#11
laceywilliams12 Registered User Join Date: Nov 2012 Posts: 71	robots.txt is a set of instructions to let the search engines know where and what to look for on your site.

01-05-2013, 02:10 AM	#12
sher.winchan58 Registered User Join Date: Jul 2012 Posts: 102	Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. It works likes this: a robot wants to vists a Web site URL, say http://www.example.com/welcome.html. Before it does so, it firsts checks for http://www.example.com/robots.txt, and finds: User-agent: * Disallow: / The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site. There are two important considerations when using /robots.txt: robots can ignore your /robots.txt. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention. the /robots.txt file is a publicly available file. Anyone can see what sections of your server you don't want robots to use. __________________ pv installers \| green deal

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)