Robots txt
 

What is the role of robots file..?

If you don't want to show any folder , pictures particular any thing than you have to use code .

user-agent*
Disalllow http://www.snapdeal.com/thumbals

If you don't want to cache index any page to search engine crawler then your can use robot.txt

Robot.txt is an on-page SEO technique and it is basically used to allow for the web robots also known as the web wanderers, crawlers or spiders. It is a program that traverses the website automatically and this helps the popular search engine like Google to index the website and its content.

Robot.txt is a simple txt file which are use for controlling Search engine's bot to crawl specific pages. you can be restricted search engine bot to crawl the any page by using robot.txt.

Robot.txt is used for hiding the website page from search engines.

Through Robot.txt file we can hide unwated pages of our webiste at the time of crawling.

Robot.txt is a text file that allow or dis allow crawlers to go through or not.

it helps to hide the secure webpage.

Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site

If you dont want to show your website pages then use robot.txt.

Robots text will prevent crawling the some pages which you don't want to show

Robot.txt is a text file that tells the search engine which pages should not crawl and index by search engine in that domain

Robot.txt help your page not cache index to search engine.

Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site

A file tell web bots what pages and directories to index or not index.

robots.txt file assist the search engine in crawling.

The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.

Robot.txt is an on-page SEO technique and it is basically used to allow for the web robots also known as the web wanderers, crawlers or spiders. It is a program that traverses the website automatically and this helps the popular search engine like Google to index the website and its content.

Allow and disallow the google bot to crawl the website pages.