robots.txt
 

why we use robots.txt file??

Content of this file tells search engine do not crawl that specific area and don't want indexed by any search engine.

Robots.txt file tells search engine crawlers how they should visit your site. If there are files and directories you do not want indexed by search engines, you can use a robots.txt file to define where the robots should not go. These files are very simple text files that are placed on your web server.

Hello,
Really this is nice answer and i am satisfied with you but one more question is find in my mind that, I want to crawled page on search engine only some time.
Please give me reply thanks.

to permit google crawl to spesific page...

thanks for sharing your view, and make clear to me use of this file.

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site .