|
The robots.txt file is a text file used in SEO . It is intended for search engine indexing robots to specify which pages can or can not be indexed. So any search engine starts exploring a website by looking for the robots.txt file at the root of the site.
Despite this, the importance of the robots.txt file is sometimes underestimated. Here, however, is important information on this subject. A Google employee has just indicated that if your site has a robots.txt file , and that it is not readable by Google (so if the returned http code is neither 200 nor 404), then the robot will simply stop crawling the site.
Do not forget to check if your file returns code 200. Note that if you do not have one, it will not pose any specific problem (code 404). Your site will be fully readable by all the different robots of different engines.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
,
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
,
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
,
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|