What is robot text
 

Do you know about robot text? please say and discuss about it.

Robots.txt is a file through which you can guide search engines to crawl or not to crawl certain sections of your website.Robot.txt is used to prevent crawlers crawl on a certain page.

By the help of robots.txt file, search engines crawls your webpages or not crawls your pages.
day picnic around delhi

Robot.txt is an on-page SEO technique and it is basically used to allow for the web robots also known as the web wanderers, crawlers or spiders. It is a program that traverses the website automatically and this helps the popular search engine like Google to index the website and its content.

Just create a file named robots.txt, put it on your root directory of your website(where index.htm/index.php is), edit with a text editor and add these:

User-agent: *
Disallow: *something*

robots.txt is a text file which instructs the search engine what are the pages should be crawled.

Robots.txt is the text file that is mostly used to instruct search engine which page should be crawled and which shouldn't be crawled.

Robots.txt is a file that you can add to your website which instructs the search engine crawler which pages to crawl. You can also instruct the crawlers not to crawl some pages through robots.txt.

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit.

The structure of a robots.txt is pretty simple (and barely flexible) – it is an endless list of user agents and disallowed files and directories. Basically, the syntax is as follows:

User-agent:

Disallow:

“User-agent” are search engines' crawlers and disallow: lists the files and directories to be excluded from indexing. In addition to “user-agent:” and “disallow:” entries, you can include comment lines – just put the # sign at the beginning of the line:

# All user agents are disallowed to see the /temp directory.

User-agent: *

Disallow: /temp/

Robots.txt file is in the root directory of a website which has been created to direct
the activity of search engine crawlers or spiders.

The concept and structure of robots.txt has been developed more than a decade ago. Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. The location of robots.txt is very important. It must be in the main directory because otherwise user agents (search engines) will not be able to find it – they do not search the whole site for a file named robots.txt.

Go on......Thanks all..

as you can easily judge by its name, robot txt means message or txt for robot (search engine spiders). In you webmaster tell bots which page to crawl or which not.

I do not see anymore effects of Robot.text file. Search Engine also index those one which comes under no follow. I tried that. No effect of this file now.

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.

robots text is normal file in which you can give instruction which page you crawl or not to all robots of search engines.

Robots.txt is only using for security purpose.It will not allow an anonymous users frequently into the website.

Robots.txt, is a text file present in the root directory of a website. The Robots.txt file is a convention created to direct the activity of search engine crawlers or web spiders. The file tells the search engine crawlers which parts to web and which parts to leave alone in a website, differing between what is viewable to the public and what is viewable to the creators of the website alone.

Robots .txt file is the place where you can write rules for web spiders about how to crawl the site and which pages should be excluded. You can write rules specifically for each web spider.

Hi,

With the help of robots.txt file search engines crawls your webpages.......

Thanks
Team from
kryptonsoft

Robots file is a simple text file which is uploaded on root directory this is the file which is crawled by the crawler very first. So in this file we write the code which file we want to block from the crawler and other one we want to crawl.

Main use of this file is just to block the admin section which generally we wish do not crawl by the search engine.

Hope you understood.

Hello,

With the help of robots.txt file search engines crawls your webpages.......

Thanks
Team from
Kryptonsoft

iPhone App Development
iPhone Games Development
Offshore iPhone Development

It is a text file which tells to search engine robots that which web page you would not like to visit. It is one kind of file or folder which tells search engine to avoid the operations described in robot.txt file.

Here is some information about Robot text.

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.

It works like this: a robot wants to visit a Web site URL, say http://www.example.com/welcome.html. Before it does so, it firsts checks for http://www.example.com/robots.txt, and finds:

User-agent: *
Disallow: /

The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.
There are two important considerations when using /robots.txt:

• Robots can ignore your /robots.txt. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention.
• The /robots.txt file is a publicly available file. Anyone can see what sections of your server you don't want robots to use.

So don't try to use /robots.txt to hide information.

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit.

Robot.txt is a file that instructs crawlers and bots whether to crawl a page or not. If yes then which page, etc.

You can allow or disallow bots from crawling your pages.

Robots.txt is a file through which you can guide search engines to crawl or not to crawl certain sections of your website.Robot.txt is used to prevent crawlers crawl on a certain page..

The robots exclusion standard, also known more commonly as Robots.txt, is a text file present in the root directory of a website. The Robots.txt file is a convention created to direct the activity of search engine crawlers or web spiders. The file tells the search engine crawlers which parts to web and which parts to leave alone in a website, differing between what is viewable to the public and what is viewable to the creators of the website alone. A Robots.txt file is frequently used by search engines to categorize and archive web pages, or by webmasters to proofread source codes.

Its simply a text file Which is used to instruct the Search engine which files to be crawl or which not.

The robots exclusion standard, also known more commonly as Robots.txt, is a text file present in the root directory of a website. The Robots.txt file is a convention created to direct the activity of search engine crawlers or web spiders. The file tells the search engine crawlers which parts to web and which parts to leave alone in a website, differing between what is viewable to the public and what is viewable to the creators of the website alone. A Robots.txt file is frequently used by search engines to categorize and archive web pages, or by webmasters to proofread source codes.

nice dude.....thank 4 all.......

using Robots.txt is a file we can guide search engines to crawl or not to crawl certain sections of your website.

Actually the search engines handle nofollow differently. Google will not follow the link, assign PR to it, or give it weight in its algorithm. Yahoo will follow the link but give it no weight in their algorithm. MSN says they won't give the link any value but it is hard to confirm exactly what they're doing.

Well ... robots.text isa text file in which you difine instruction which page you want to be crawl or not to all robots of search engines.

Robots.txt is a text file you put it on your web site to tell search robots which pages you would like them not to visit. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site.