What is robots.txt?
 

This is a page that gives search engines information about the pages a company wants indexed or crawled. You can find this page by doing to YOURDOMAIN/robots.txt.

Robots.txt file is very helpful to disallow the pages which you don't want to index in the search engine.

picnic spots near delhi | water park in delhi

Its a file to prevent the crawlers to not to crawl the restricted pages in your site.

Robot.txt file is use for hide web page from search engine, if you don't want to index any page of website you can use this robot.txt and disallow that page....

with help to robots.txt Google find website internal page easily

its restrict the robots not crawler your all page only index page they can craw.

thanx guys, for giving that nice suggestion or answer regarding robot.txt.

The Robot Different Frequent, additionally known as the Robots Exclusion Protocol or robots.txt method, is a personalized to training working net elephants as well as added net spiders about acquiring all or portion of a site that's otherwise readily readable. Bots are usually employed by yahoo to truly save and classify websites, or by web masters to estimate source-code. The normal differs from, but could be used together with, Site Maps, a robot intro common for web sites.

The robots.txt is a simple text file in your web site that inform search engine bots how to crawl and index website or web pages.

Robots.txt file is a system file. If you still want to common things it by your own, create a text file using notepad or any other text editor and name it robots.txt. Then upload it to the root folder of your site by means File Manager or FTP.

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.

Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.