|
The robots.txt file
The robots.txt not command, but a agreement. Search engine in the grab a web site page, will be the first to grab the robots.txt file, so the file the meaning of existence is tell search engines what page can be grab, which page cannot be grab. When the search engine grab site web page, it will first visit the web site under the root directory of the robots files, if there is the documents, search engine will be defined in the paper the content of grab, if not, the search engines will grab all allow it can catch page.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|