RH-Calvin

Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.

__________________
█ Cheap VPS | $1 VPS Hosting
█ Cheap Windows VPS | Windows Server with Remote Desktop Access
█ Cheap Dedicated Server | Servers with Free IPMI Setup

albertc99

The robots.txt file is a text file used in SEO . It is intended for search engine indexing robots to specify which pages can or can not be indexed. So any search engine starts exploring a website by looking for the robots.txt file at the root of the site.

Despite this, the importance of the robots.txt file is sometimes underestimated. Here, however, is important information on this subject. A Google employee has just indicated that if your site has a robots.txt file , and that it is not readable by Google (so if the returned http code is neither 200 nor 404), then the robot will simply stop crawling the site.

Do not forget to check if your file returns code 200. Note that if you do not have one, it will not pose any specific problem (code 404). Your site will be fully readable by all the different robots of different engines.

__________________
Rajasthan Tours, Varanasi Tours, India Tour Packages, Lodhi Garden in Delhi

adibitar69

Robots.txt is a file in website which gives instruction to search engine crawler which page of website should be indexed and which Web pages should be ignored.

__________________
Android app development companies Dubai | Best Mobile Apps Development Companies

rajkiran

Robots.txt is a text file located in the site's root directory for search engines Spiders and Crawlers what website pages and files you want or don't want them to visit.

Zora2012

Robots.txt" is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site, or at all.

__________________
Shopping Cart Development - Shopping Cart- PrestaShop, Zencart; CMS - WordPress, Joomla; Custom Development

Link Popularity Building - SEO, Pay Per Click (PPC), SEM, Link Popularity Building, SEO Copywriting, SMO, SMM

vinukum

Robots.txt is a text file webmasters create to instruct web robots how to crawl pages on their website. The robots.txt file is part of the the robots exclusion protocol (REP), a group of web standards that regulate how robots crawl the web, access and index content, and serve that content up to users.
Basic format of the robot.txt file.
User-agent: [user-agent name]
Disallow: [URL string not to be crawled]

__________________
Andhra style Restaurants in Bangalore | Order Andhra Food Online in Bangalore | Home Delivery Restaurants Bangalore | Order Andhra Biryani Online

steveamith

IF you want to show your page to the user but do not want to google crawl it then we use Robot.txt.