Robots.txt combined with vBSEO necessary?

I've been reading around on various vbulletin forums and found this post on vBulletin setup his third tip about restricting search engine crawling:

Code:

User-agent: *Disallow: /forums/ajax.phpDisallow: /forums/attachment.phpDisallow: /forums/calendar.phpDisallow: /forums/cron.phpDisallow: /forums/editpost.phpDisallow: /forums/global.phpDisallow: /forums/image.phpDisallow: /forums/inlinemod.phpDisallow: /forums/joinrequests.phpDisallow: /forums/login.phpDisallow: /forums/member.phpDisallow: /forums/memberlist.phpDisallow: /forums/misc.phpDisallow: /forums/moderator.phpDisallow: /forums/newattachment.phpDisallow: /forums/newreply.phpDisallow: /forums/newthread.phpDisallow: /forums/online.phpDisallow: /forums/poll.phpDisallow: /forums/postings.phpDisallow: /forums/printthread.phpDisallow: /forums/private.phpDisallow: /forums/profile.phpDisallow: /forums/register.phpDisallow: /forums/report.phpDisallow: /forums/reputation.phpDisallow: /forums/search.phpDisallow: /forums/sendmessage.phpDisallow: /forums/showgroups.phpDisallow: /forums/subscription.phpDisallow: /forums/threadrate.phpDisallow: /forums/usercp.phpDisallow: /forums/usernote.php

Do you think this is necessary?

A robots.txt is absolutely recommended to keep bots from trying to index pages you do not want it too.

So would you recommend what I posted above or is there something else you might recommend?

It's different for every site.

Disallow printthread.php, showpost.php, newthread.php, newreply.php, poll.php (make sure you aren't rewriting these urls)

Basically everything but your forums and threads. I even recommend disallowing the archive myself.

Originally Posted by BamaStangGuy

It's different for every site.

Disallow printthread.php, showpost.php, newthread.php, newreply.php, poll.php (make sure you aren't rewriting these urls)

Basically everything but your forums and threads. I even recommend disallowing the archive myself.

Well, with the standard CRR's does that affect the urls you listed?

Hmmm, I wonder why this isn't mentioned in the vBSEO installation steps or something?

Originally Posted by ChrisGuthrie.net

Hmmm, I wonder why this isn't mentioned in the vBSEO installation steps or something?

Because it is your own decision to use a robots.txt or not

Hi Bama,

May i know which setting is for disabling newthread.php, newreply.php in VBSEOCP?

Thank you
Edy

Originally Posted by kellogs

Hi Bama,

May i know which setting is for disabling newthread.php, newreply.php in VBSEOCP?

Thank you
Edy

You shall disable newthread.php and newreply.php within a robots.txt file.