Robots.txt is a text (not html) files which search robots in your site which pages are not visited. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site.

The robots.txt is a file that follow the search engine instruction.

The robots.txt file is a text file that tells search engine crawlers which portions of your website they should NOT index. If you don't want to restrict search engine crawlers, you should simply create an empty robots.txt file (e.g., touch robots.txt) or one that looks like this:

User-agent: *
Disallow:
Once you have created a robots.txt file, you store it in the root directory of your Web server.

Hope this helps you!!

Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit.

The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.

The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.

Robots.txt is a convention to prevent cooperating web crawlers and other web robots from accessing all or part of a website which is otherwise publicly viewable.

The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.

Robot.txt is a simple notepad file through which crawler crawl your website.

Robots.txt is a text file which gives instruction to the search engine crawler to which page has to be crawled or not by allowing or disallowing.

The robots.txt is a simple text file in your web site that informs search engine bots how to crawl and index website or web pages. It is great when search engines frequently visit your site and index your content but often there are cases when indexing parts of your online content is not what you want.

to block unwanted files we will use this record

The importance of adding custom robots.txt to blogger

Blogger comes with a default robots.txt file. However, blogger allows us to customize this file to suit our needs. This is called custom robots.txt file and I will show you the relevance of customizing this file and how to add it to the blogger server.

What is Robots.txt file ?
A robots.txt is a code which instructs web crawlers on how to go about indexing specific pages on search engines. Before web crawlers index web pages, it will first look at this file in order to ascertain the specific instructions required of it to carry out.
So basically, a robots.txt file does two things :
1. It allows search engines to discover contents on the web
2. It allows specific contents to be indexed on search engines and served to those looking for information

A robots.txt file is usually found by default on website or blog host servers like blogger. In this tutorial, I will explain what each line of code of a robots.txt file represents and how to customize it.I will use a blogger's default robots file to illustrate the working principles of the robots.txt.
The code is as shown below:

http://www.chrisdiary.com/2017/07/th...ng-custom.html

The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. Robots are often used by search engines to categorize websites.

Robots.txt are the files that will tell the crawler which pages of the website to crawl and which pages of the website not to crawl.