|
Robots.txt is a text file that tells web robots which pages on your site to crawl.
Ex of crawl all pages on website.
User-agent: *
Disallow:
It also tells web robots which pages not to crawl.
Ex of not to crawl about us page
User-agent: *
Disallow:/aboutus
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|