What is the limit of a robot.txt file? - Site Owners Forums

kaurksifat · 07-18-2018, 06:43 AM

Yes, there is a limit to robots.txt file in terms of filesize 500 KB as Vipin points out.

A maximum file size may be enforced per crawler. Content which is after the maximum file size may be ignored. Google currently enforces a size limit of 500kb.

Source: Robots.txt Specifications

In terms of limitations (not sure if you meant that as well), there’s the fact that the robots.txt file is merely a optional directive, not a mandate. So search engines don’t have to adhere to your guidelines. It’s unethical not to, but it’s not illegal.

We’ve actually covered this queston in our FAQ here: Robots.txt file: the ultimate reference guide. You can read a lot more on the robots.txt file if you’re interested in learning more.

Learn Digital Marketing in Gurgaon top digital marketing courses

07-18-2018, 06:43 AM	#1
kaurksifat Registered User Join Date: May 2018 Posts: 31	Yes, there is a limit to robots.txt file in terms of filesize 500 KB as Vipin points out. A maximum file size may be enforced per crawler. Content which is after the maximum file size may be ignored. Google currently enforces a size limit of 500kb. Source: Robots.txt Specifications In terms of limitations (not sure if you meant that as well), there’s the fact that the robots.txt file is merely a optional directive, not a mandate. So search engines don’t have to adhere to your guidelines. It’s unethical not to, but it’s not illegal. We’ve actually covered this queston in our FAQ here: Robots.txt file: the ultimate reference guide. You can read a lot more on the robots.txt file if you’re interested in learning more. Learn Digital Marketing in Gurgaon top digital marketing courses __________________ Hi, I am Sifat Arora working as digital marketing Consulatnt for BridgeABLE.in. They are providing Digital Marketing Training in Gurgaon. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. ,

Thread Tools
Show Printable Version Email this Page
Display Modes	Rate This Thread
Switch to Linear Mode Hybrid Mode Switch to Threaded Mode	Rate This Thread:

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
What is the limit of a robot.txt file?	artijain1900	Search Engine Optimization	11	05-17-2022 12:48 AM
What is the limit of a robot.txt file?	ruhikhan	Search Engine Optimization	7	01-22-2018 04:28 AM

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)