robots.txt suggestions

I am trying to figure out a few things regarding the robots.txt file. Mine currently looks like this:

# Allow Archiver
User-agent: ia_archiver

Allow: /User-agent: Slurp
Crawl-delay: 60

User-agent: *

Disallow: /includes/
Disallow: /install/
Disallow: /customavatars/

What is the purpose of disallowing all of the vB directories as discussed in this thread? Also, why would I want to disallow individual file types (i.e. .php, .html, etc.)?

Thank you,

Does anyone have any information regarding this subject? I want to make sure the robots.txt file is properly setup. I have seen people disallow every folder within vBulletin but I don't know why they would do that. Same for the individual file types.

Thank you for any and all information you are able to provide.

Can anyone with experience fill us in?

I would like this information as well. A sticky should be made with how to setup robots.txt.