What is robots.txt?
 

It is a kind of text file utilized to provide the instructions to the crawlers about the caching and indexing of a website, directory, domain or a file of a webpage.

You have covered up the basic definition of what it means

Robots.txt helps to index ourwebsite all pages on google.

Robots.txt files are used to inform the crawler which URL to crawl and which to avoid.

Robots.txt files are used to inform the search engine crawler to which URL to allow search engines and which is to disallow .

Crawling of website by robots is first step to getting ranked in SearchEngines. The Robot Exclusion Standard, also known as the Robots Exclusion Protocol or robots.txt protocol. Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.

By way of example, my website is css and html based, which means that it is not currently built with a CMS such as WordPress' and I use the robot.txt file to excule search engines from reaching the development directories I use to test drive CMS softwares such as WordPress. When my WordPress based site is ready for being seen by the world, I will remove the exclusion line from my robot.txt.

You can find the syntax supported by robot.txt files by doing a google search. They will also tell it where to put it on your site, which is the same directory where your index.html file is. If you use a CMS like WordPress, I believe the software generates the robot.txt file according to the parameters you choose within WordPress, but I'm less familiar with these than with the css and html based web sites.

Robot.txt is a text file which you put in your website to tell search engine which pages you would like not to crawl.
For an Example:

User Agest: *

Disallow:/admin/
Disallow:/epe/de
Disallow:/dwsdf/fdedx/

Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit.

Robots.txt is a text file for gives the intruction of bots. particular search engine bots or all search engine bots which file or folder have cached or not cached.

robots.txt is a text file that search engine information allow or not.

Its pretty much a file that search engines look at to figure out whether to index your site or not.

it is a .txt file, that doesn't crawled by the web spider.

thanks for the information........

its the text file which with the name robot.txt use to disallow the crawler to crawl the page that we don't need to crawl