Site Owners Forums - Webmaster Forums - View Single Post - How to prevent particular pages from crawler?

brainmoore · 11-13-2014, 10:47 AM

There are many ways to prevent pages from getting crawled by bots by using some codes on the website. For example: one is mentioned above, other we can use robots.txt file, we can server level meta tags which can tell bots not to crawl or index any page etc..

11-13-2014, 10:47 AM	#3
brainmoore Registered User Join Date: Oct 2014 Posts: 32	There are many ways to prevent pages from getting crawled by bots by using some codes on the website. For example: one is mentioned above, other we can use robots.txt file, we can server level meta tags which can tell bots not to crawl or index any page etc..