What is robot.txt file and what are the benefits of using it? - Page 3 - Site Owners Forums

luffy268 · 12-19-2016, 03:11 AM

Robots.txt is a text file which has few lines and this file interacts with all kind of crawlers or spiders like Googlebot, which is Google search engine's spider.

Search Engine always want to index good and fresh contain on the web and these robot.txt instruct the web crawlers to how to index and crawl your blog in the search results.

Remember,Crawler or spider will firstly look at your robots.txt files to obey the rules you have instructed It's mean you can restrict any web page on your blog from web crawlers so that it can’t get indexed in search engines like your blog your demo page, labels page, Achieve or any other pages that are not as important to get indexed.

Nandu41 · 12-21-2016, 02:46 AM

Robots.txt is a text file which has few lines and this file interacts with all kind of crawlers or spiders like Googlebot, which is Google search engine's spider.

Search Engine always want to index good and fresh contain on the web and these robot.txt instruct the web crawlers to how to index and crawl your blog in the search results.

12-19-2016, 03:11 AM	#31
luffy268 Registered User Join Date: Dec 2016 Posts: 94	Robots.txt is a text file which has few lines and this file interacts with all kind of crawlers or spiders like Googlebot, which is Google search engine's spider. Search Engine always want to index good and fresh contain on the web and these robot.txt instruct the web crawlers to how to index and crawl your blog in the search results. Remember,Crawler or spider will firstly look at your robots.txt files to obey the rules you have instructed It's mean you can restrict any web page on your blog from web crawlers so that it can’t get indexed in search engines like your blog your demo page, labels page, Achieve or any other pages that are not as important to get indexed. __________________ *hakireview\|hakireview.com\|http://hakireview.com*

12-21-2016, 02:46 AM	#32
Nandu41 Registered User Join Date: Sep 2016 Posts: 65	Robots.txt is a text file which has few lines and this file interacts with all kind of crawlers or spiders like Googlebot, which is Google search engine's spider. Search Engine always want to index good and fresh contain on the web and these robot.txt instruct the web crawlers to how to index and crawl your blog in the search results. __________________ web designing company in hyderabad \| web designing company india \|sap fico online training

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)