|
Robots.txt is basically a text file which is designed for machines, robots which crawl websites online. It uses some codes which tells bots whether to crawl the website/page or not, it is more like a permission given area.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|