[jamesBA]

here are two important considerations when using /robots.txt:

1- robots can ignore your /robots.txt. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention.
2-the /robots.txt file is a publicly available file. Anyone can see what sections of your server you don't want robots to use