Thread: What is the use of Robots.txt?
View Single Post
Old 02-17-2016, 11:31 PM   #8
cairominerals
Registered User
 
Join Date: Feb 2016
Location: Egypt
Posts: 15
It is a text file. It instruct the crawler which page has crawl or not
cairominerals is offline   Reply With Quote