Robots.txt is only using for security purpose.It will not allow an anonymous users frequently into the website.

Robots.txt, is a text file present in the root directory of a website. The Robots.txt file is a convention created to direct the activity of search engine crawlers or web spiders. The file tells the search engine crawlers which parts to web and which parts to leave alone in a website, differing between what is viewable to the public and what is viewable to the creators of the website alone.

Robots .txt file is the place where you can write rules for web spiders about how to crawl the site and which pages should be excluded. You can write rules specifically for each web spider.

Hi,

With the help of robots.txt file search engines crawls your webpages.......

Thanks
Team from
kryptonsoft

Robots file is a simple text file which is uploaded on root directory this is the file which is crawled by the crawler very first. So in this file we write the code which file we want to block from the crawler and other one we want to crawl.

Main use of this file is just to block the admin section which generally we wish do not crawl by the search engine.

Hope you understood.

Hello,

With the help of robots.txt file search engines crawls your webpages.......

Thanks
Team from
Kryptonsoft

iPhone App Development
iPhone Games Development
Offshore iPhone Development

It is a text file which tells to search engine robots that which web page you would not like to visit. It is one kind of file or folder which tells search engine to avoid the operations described in robot.txt file.

Here is some information about Robot text.

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.

It works like this: a robot wants to visit a Web site URL, say http://www.example.com/welcome.html. Before it does so, it firsts checks for http://www.example.com/robots.txt, and finds:

User-agent: *
Disallow: /

The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.
There are two important considerations when using /robots.txt:

• Robots can ignore your /robots.txt. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention.
• The /robots.txt file is a publicly available file. Anyone can see what sections of your server you don't want robots to use.

So don't try to use /robots.txt to hide information.

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit.

Robot.txt is a file that instructs crawlers and bots whether to crawl a page or not. If yes then which page, etc.

You can allow or disallow bots from crawling your pages.

Robots.txt is a file through which you can guide search engines to crawl or not to crawl certain sections of your website.Robot.txt is used to prevent crawlers crawl on a certain page..

The robots exclusion standard, also known more commonly as Robots.txt, is a text file present in the root directory of a website. The Robots.txt file is a convention created to direct the activity of search engine crawlers or web spiders. The file tells the search engine crawlers which parts to web and which parts to leave alone in a website, differing between what is viewable to the public and what is viewable to the creators of the website alone. A Robots.txt file is frequently used by search engines to categorize and archive web pages, or by webmasters to proofread source codes.

Its simply a text file Which is used to instruct the Search engine which files to be crawl or which not.

The robots exclusion standard, also known more commonly as Robots.txt, is a text file present in the root directory of a website. The Robots.txt file is a convention created to direct the activity of search engine crawlers or web spiders. The file tells the search engine crawlers which parts to web and which parts to leave alone in a website, differing between what is viewable to the public and what is viewable to the creators of the website alone. A Robots.txt file is frequently used by search engines to categorize and archive web pages, or by webmasters to proofread source codes.

nice dude.....thank 4 all.......