What is the importance of robot.txt in SEO? - Site Owners Forums

albertc99 · 02-06-2019, 10:23 PM

The robots.txt file is a text file used in SEO . It is intended for search engine indexing robots to specify which pages can or can not be indexed. So any search engine starts exploring a website by looking for the robots.txt file at the root of the site.

Despite this, the importance of the robots.txt file is sometimes underestimated. Here, however, is important information on this subject. A Google employee has just indicated that if your site has a robots.txt file , and that it is not readable by Google (so if the returned http code is neither 200 nor 404), then the robot will simply stop crawling the site.

Do not forget to check if your file returns code 200. Note that if you do not have one, it will not pose any specific problem (code 404). Your site will be fully readable by all the different robots of different engines.

02-06-2019, 10:23 PM	#3
albertc99 Registered User Join Date: Jan 2016 Posts: 49	The robots.txt file is a text file used in SEO . It is intended for search engine indexing robots to specify which pages can or can not be indexed. So any search engine starts exploring a website by looking for the robots.txt file at the root of the site. Despite this, the importance of the robots.txt file is sometimes underestimated. Here, however, is important information on this subject. A Google employee has just indicated that if your site has a robots.txt file , and that it is not readable by Google (so if the returned http code is neither 200 nor 404), then the robot will simply stop crawling the site. Do not forget to check if your file returns code 200. Note that if you do not have one, it will not pose any specific problem (code 404). Your site will be fully readable by all the different robots of different engines. __________________ *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* , *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* , *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* , *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.*

Thread Tools
Show Printable Version Email this Page
Display Modes	Rate This Thread
Switch to Linear Mode Switch to Hybrid Mode Threaded Mode	Rate This Thread:

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
robot.txt	Shivam Tripathi	Web Promotion	5	12-16-2018 10:11 PM
What is Robot.txt?	kumar1_16	General Discussion	11	03-29-2017 03:15 AM
how to keyword importance in seo?	ruhanikakhan	Search Engine Optimization	19	06-22-2016 09:14 AM
Importance of sub domain.	Adhitimishra23	Traffic Corner	1	12-24-2015 09:12 PM
Does Anyone Know About The Importance Of SEO?	brettlee585	Search Engine Optimization	5	11-26-2012 05:15 AM

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)