Site Owners Forums - Webmaster Forums - View Single Post

rajkiran · 03-05-2019, 04:19 AM

A crawler is a program that visits Web sites and reads their pages and other information in order to create entries for a search engine index. The major search engines on the Web all have such a program, which is also known as a "spider" or a "bot."

03-05-2019, 04:19 AM	#3
rajkiran Registered User Join Date: Aug 2016 Posts: 961	A crawler is a program that visits Web sites and reads their pages and other information in order to create entries for a search engine index. The major search engines on the Web all have such a program, which is also known as a "spider" or a "bot."