Robots txt
 

What is the role of robots file..?

If you don't want to show any folder , pictures particular any thing than you have to use code .

user-agent*
Disalllow http://www.snapdeal.com/thumbals

If you don't want to cache index any page to search engine crawler then your can use robot.txt

Robot.txt is an on-page SEO technique and it is basically used to allow for the web robots also known as the web wanderers, crawlers or spiders. It is a program that traverses the website automatically and this helps the popular search engine like Google to index the website and its content.

Robot.txt is a simple txt file which are use for controlling Search engine's bot to crawl specific pages. you can be restricted search engine bot to crawl the any page by using robot.txt.

Robot.txt is used for hiding the website page from search engines.

Through Robot.txt file we can hide unwated pages of our webiste at the time of crawling.

Robot.txt is a text file that allow or dis allow crawlers to go through or not.

it helps to hide the secure webpage.

Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site

If you dont want to show your website pages then use robot.txt.

Robots text will prevent crawling the some pages which you don't want to show

Robot.txt is a text file that tells the search engine which pages should not crawl and index by search engine in that domain

Robot.txt help your page not cache index to search engine.

Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site