|
Some websites stop web crawlers from visiting them. These pages will be left out of the index, along with pages that no-one links to. The information that the web crawler puts together is then used by search engines. It becomes the search engine's index.
|