|
Googlebot (or any search engine spider) crawls the web to process information. Until Google is able to capture the web through osmosis, this discovery phase will always be essential. Google, based on data generated during crawl time discovery, sorts and analyzes URLs in real time to make indexation decisions
|