What is robots.txt?
 

Robots.txt is a text file used to give instructions to the search engine crawlers about the caching and indexing of a webpage, domain, directory or a file of a website.

The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) on how to crawl & index pages on their website.

Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.

Robots.txt is a convention to prevent cooperating web crawlers and other web robots from accessing all or part of a website which is otherwise publicly viewable.

Robots.txt files inform search engine spiders how to interact with indexing your content. If you do not have a robots.txt file, your server logs will return 404 errors whenever a bot tries to access your robots.txt file.

This is very useful post. Thank you.........

Thanks for share this information.

The robots.txt is a simple text file in your web site that inform search engine bots how to crawl and index website or web pages.

The robots.txt is a simple text file in your web site that informs search engine bots how to crawl and index website or web pages. It is great when search engines frequently visit your site and index your content but often there are cases when indexing parts of your online content is not what you want.

Robots.txt is text file that holding the information for search engine crawler about caching and indexing of webpage, Domain..etc

Robots.txt is a text file and having the information for dofollow or no-follow for the crawlers to crawl a website web pages, and providing them indexing.

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note �Please, do not enter� on an unlocked door � e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter.

robot.txt tells the search engine which data you want to crawl and which is not.

Robot.txt is a simple notepad file through which crawler crawl your website.

Robot.txt file indicate search crawler which information in this website is prohibited or do not crawl. if owner of the site does not want any of the file to be indexed or crawled by robot or spider then robot.txt file being used