Thread: how to work robot txt and crawl
View Single Post
Old 02-26-2015, 10:31 PM   #3
James_Tarrier
Registered User
 
Join Date: Nov 2014
Location: UK
Posts: 26
Robot.txt is a file to show way to crawler which page is index in their database.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
James_Tarrier is offline   Reply With Quote