Site Owners Forums - Webmaster Forums - View Single Post - Why Is Important Of Xml and robots.txt For A Website?

saleoid · 08-08-2014, 04:22 AM

XML is containing all sites links and Search engine will use to crawl and index all links from that XML. It really help in indexing all your website's links.

While robots.txt is used to tell search engines crawler that which page they should crawl or which page they should not crawl.

08-08-2014, 04:22 AM	#3
saleoid Registered User Join Date: Aug 2014 Posts: 14	XML is containing all sites links and Search engine will use to crawl and index all links from that XML. It really help in indexing all your website's links. While robots.txt is used to tell search engines crawler that which page they should crawl or which page they should not crawl.