|
Robots.txt is the root file of the website, where you mention which pages of the website are allowed to be crawled and which pages are blocked.
If your website is not having Robots.txt file, than by default all pages of the website are allowed to be crawled.
Robots.txt tag is the substitute of Robots.txt file. It also does the same work, the only difference between Robots.txt file and tag is, tag need to be mentioned in all the pages, whereas file is uploaded only once in the root file.
|