Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.
It works likes this: a robot wants to vists a Web site URL, say http://www.example.com/welcome.html. Before it does so, it firsts checks for http://www.example.com/robots.txt, and finds:
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
- Taste Of The Wild Dog
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
– WORK VIA THE INTERNET WITH
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
-
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|