What Is Robots.txt?
 

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit.

Robots.txt file can be used to tell search engines that which of your website pages/directories should not be crawled.

Somehow you can say like this but purpose is same to disallow search engine to index the content.

Very true.Generally we use it to save unusual crawling of spider on those pages having duplicate content or files which we don't want to share publicly.
Thanks

it gives the instruction to not crawl the specific pages or file mention in txt

Robots.txt will inform the search engine what are the pages should be crawled

Robots.txt will tell the search engine what to crawl and what not to. This helps if you do not want search engines to crawl and index some of your private data.

Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site..

Thanks to share this great information with us, Robot.txt files are very important in on-page field to make our website error free.

The robots.txt is for a search engine, visit the site of the time want to see the first file. The robots.txt file told spiders in the server is what document can be looked at.
When a search spider visit a website, it will first check the site to root directory whether there is the robots.txt, if present, search according to the document robot to determine the content of the scope of the visit; If the file does not exist, all the search spiders will be able to access web site was not all password protected pages.

robot.txt file is simple text document.
1. open notepad
2. type these lines.
3. save your file as robots.txt (extension should be .txt)
4. placed this file in your root folder
5. you can view it

It is a code within the root file that acts to avoid spiders from crawling over the site or page. It is a technique used in SEO to avoid indexing.

Robot.txt is the way to stop the search engines to crawl any page of the website. This is required because some time there is private data on some pages of the website and the website owner does not want search engines to crawl those pages and read that useful information. In those cases robot.txt is the file in which all the pages that we don't want to crawl are added and this file is then added to the root folder of the website.

Robots.txt is a file through which you can guide search engines to crawl or not to crawl certain sections of your website.

Thanks and Regards