[DanielGarneau]

By way of example, my website is css and html based, which means that it is not currently built with a CMS such as WordPress' and I use the robot.txt file to excule search engines from reaching the development directories I use to test drive CMS softwares such as WordPress. When my WordPress based site is ready for being seen by the world, I will remove the exclusion line from my robot.txt.

You can find the syntax supported by robot.txt files by doing a google search. They will also tell it where to put it on your site, which is the same directory where your index.html file is. If you use a CMS like WordPress, I believe the software generates the robot.txt file according to the parameters you choose within WordPress, but I'm less familiar with these than with the css and html based web sites.