Site Owners Forums - Webmaster Forums - View Single Post

GaryKoster · 07-15-2014, 06:50 AM

A crawler is a program that visits all websites and their pages and checks that they are ethical or not and create a list for indexing in Google. They are also called Google bot or spider.

07-15-2014, 06:50 AM	#5
GaryKoster Registered User Join Date: Jul 2014 Location: USA, indiana Posts: 38	A crawler is a program that visits all websites and their pages and checks that they are ethical or not and create a list for indexing in Google. They are also called Google bot or spider.