I recommend you getting a moderate delay as, it will work better for you.
Here is a post on robots.txt. It is actually for a wordpress website, but it explains in detail, and should give you an idea of what you might want to do with yours.
boganmarketing.com/blog/ultimate