What is robots.txt?
 

What is robots.txt?

robot.txt is the text file its help to stop the crawler to index the page which is not allow in the robot.txt and its upload with help of the google webmaster tools and code of robot.txt look like such as the
user agent: *
Disallow /abc/
disallow /xyz/

The particular robots exclusion protocol (REP) or robots(dot) txt is usually a textual content record website owners build to instruct robots (typically Google search robots ) they way to get in addition to directory web pages on their website.

yes agree with the both that robot.txt file is used to denied the google crawler to index some some pages of the pages or guide the crawler that which has to index and which not

Robots.txt file is really useful to disallow a particular page of the site.

disallow / abc


picnic spot in delhi | camp wild dhauj

Robot.txt is a text file which inform the crawlers or spiders weather they should visit the page or not.

Robot.txt is used to give knowledge about any website to google. Search engines can know which thing has to be crawled in any website and which should not to be crawled with the help of robot.txt.

Robots.txt is a file in the root directory of your web site that instructs web crawlers what parts, or all, or none of your site they are allowed examine.

User-agent: *
Disallow: /

it is used to prevent security breaking..

Robots is the most useful file, you can control google crawling process by using robots.txt