What is the use of Robots.txt ?
 

Hello friends,


I want to know that What is the use of Robots.txt ? tell me ..

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

Web site owners use the /robots.txt file to give instructions about their site to web robots.The "Disallow: /" tells the robot that it should not visit any pages on the site.

Web site owners use the /robots.txt file to give instructions about their site to web robots.The "Disallow: /" tells the robot that it should not visit any pages on the site.

Robot.txt files are used to give information to the spiders to visit the website for crawling and indexing.We can use Disallow option which will tell the robot not to visit the website.

Hi, yes right I am agree with you.

Robots.txt is nothing hide the some pages of our site because of privacy.for example you have an e-commerce site like shopclues, flipkart in festival time we post some offer in particular page after u no need but it can use future purpose in that situation also we can use.

It's a file used by the webmasters to give instructions to web robots. It helps if we don't want any page to be crawled by any web robot.

Robots.txt is a text file,we can write in word and notepad. Robots.txt you put on your website to tell the search engine which pages you would like them not to visit.

It is great when search engines frequently visit your site and index your content but often there are cases when indexing parts of your online content is not what you want. For instance, if you have two versions of a page (one for viewing in the browser and one for printing), you'd rather have the printing version excluded from crawling, otherwise you risk being imposed a duplicate content penalty. Also, if you happen to have sensitive data on your site that you do not want the world to see, you will also prefer that search engines do not index these pages (although in this case the only sure way for not indexing sensitive data is to keep it offline on a separate machine). Additionally, if you want to save some bandwidth by excluding images, stylesheets and javascript from indexing, you also need a way to tell spiders to keep away from these items.

Web site owners use the /robots.txt file to give instructions about their site to web robots.The "Disallow: /" tells the robot that it should not visit any pages on the site that is called for use robot.txt

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

robots.txt is a ON PAGE SEO TECHNIQUE is used to hide your data during the google reading on your website.
it has two choice Allow and Disallow if give ur data Allow site google read your data otherwise it can't read your data

When a search engine crawls (visits) your website, the first thing it looks for is your robots.txt file. This file tells search engines what they should and should not index.