|
A robots.txt file offers directions to web robots about the pages the site owner doesn't wish to be 'Crawled'. Case in point, in the event that you didn't need your pictures to be recorded by Google and other web indexes, you'd square them utilizing your robots.txt file.
|