Robots.txt Help

Would anyone be kind enough to help me generate a correct robots.txt file. My site in regards to vBSEO setup is the same as here (everything in root, however, re-written to make forums appear to be in their own directory)

http://www.commanderssv.com/robots.txt (right now I have it just like it is here)

Thanks so much in advance!

Robots file that's should you make it by yourself, That's what you want to allow to index and what not, However I would suggest also Adding your sitemap into robots file.

Code:

Sitemap: http://domain.com/sitemap_index.xml.gz

Change domain.com with your site domain name.

Thanks, the only reason I asked is cause I really don't know what is best to allow them to crawl and what is not, lol.

Do I just copy and paste that site map code above into the robots file or do I need anything else added? By doing this am I telling robots not to access the site map?

Thanks so much and forgive the ignorance

see: Tutorial how to make a good robots.txt file

Thanks Brian, I will take a look