What is the use of Robots.txt?
 

Hello friends,

What is the use of Robots.txt?

Website owners use the Robots.txt file to give insturctions about their site to web robots this is called The Robots Exclusions Protocol.

Hi, robot.txt file is used to give instruction to the search engine about, which page to be crawled are which not.

Upcoming events in Chandigarh, Upcoming events in Mohali, Upcoming events in Panchkula, Upcoming events in Tricity

In Simple words Robot.txt files gives instructions of your website to search engines.

Robots.txt will make search engine spider to avoid crawl the particular URL of your website.

Robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.

or

Web site owners use the robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.

Robots.txt file is used to instruct crawler which page has crawl or which page not. You can block a whole folder through this file.

It is a text file. It instruct the crawler which page has crawl or not