Why is a Robots.txt File Used?
 

Why is a Robots.txt File Used?
matrimonial sites for hindu brahmin | indian matrimonial sites in new zealand |matrimony sites in gurgaon

Robots.txt google robot allows better orientation

Use a show to gateway to for search engine bot to what to index what not to

that is use for drive search engine bot

The basic format to write a robots.txt file is

User-agent: *
Disallow: /url1/
Disallow: /url2/
Disallow: /url.html

If you want to disallow few webpages crawling from search engines then you can add in this robots.txt file