Thread: Robots txt
View Single Post
Old 07-05-2013, 04:17 AM   #7
vantrendin
Registered User
 
Join Date: Jun 2013
Posts: 110
Through Robot.txt file we can hide unwated pages of our webiste at the time of crawling.
vantrendin is offline   Reply With Quote