|
Robots .txt file is the place where you can write rules for web spiders about how to crawl the site and which pages should be excluded. You can write rules specifically for each web spider.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
Last edited by sijugk; 02-24-2014 at 01:49 PM..
|