I Need Help Please! Google and others not indexing large site.

Well, I need some help please. This may be a bit long and complicated but for the sake of having all of the needed info here goes.


Current State of Affairs:

Pro Photo Home Forums has approximately the following stats: 33,000 members, 70,000 threads and 400,000 posts. I installed vbSEO about 3 months ago along with the sitemaps. They are all working fine and have been from the beginning. The google bots regularly crawl the site and all of the sitemaps report as being downloaded properly, etc. We have a google pagerank of 5. I have a robots.txt file installed and functioning properly. Here are the contents:

# For domain: Pro Photo Home Forums
# All robots will spider the domain
# Disallow directory
User-agent: *
Disallow: /forum/admincp/
Disallow: /forum/ajax.php
Disallow: /forum/articlebot/
Disallow: /attachment.php
Disallow: /attachments/
Disallow: /forum/clientscript/
Disallow: /forum/cpstyles/
Disallow: /forum/customavatars/
Disallow: /forum/avata/
Disallow: /forum/customprofilepics/
Disallow: /forum/calendar.php
Disallow: /cgi-bin/
Disallow: /forum/cron.php
Disallow: /display.html
Disallow: /forum/editpost.php
Disallow: /forum/external.php
Disallow: /frm_attach
Disallow: /forum/includes/
Disallow: /forum/inlinemod.php
Disallow: /forum/install/
Disallow: /forum/ipinfo.php
Disallow: /forum/joinrequests.php
Disallow: /forum/login.php
Disallow: /forum/markers.xml
Disallow: /forum/member.php
Disallow: /forum/memberlist.php
Disallow: /forum/misc.php
Disallow: /forum/modcp/
Disallow: /forum/moderator.php
Disallow: /ModeratorOptions.pdf
Disallow: /forum/modules/
Disallow: /forum/newattachment.php
Disallow: /forum/newreply.php
Disallow: /forum/newthread.php
Disallow: /oldindex.html
Disallow: /forum/online.php
Disallow: /forum/payment_gateway.php
Disallow: /forum/payments.php
Disallow: /forum/poll.php
Disallow: /forum/postings.php
Disallow: /forum/private.php
Disallow: /forum/profile.php
Disallow: /forum/register.php
Disallow: /forum/report.php
Disallow: /forum/reputation.php
Disallow: /forum/search.php
Disallow: /forum/sendmessage.php
Disallow: /forum/showgroups.php
Disallow: /ssm.js
Disallow: /ssmItems.js
Disallow: /forum/subscription.php
Disallow: /subscriptions/
Disallow: /test.html
Disallow: /forum/test.php
Disallow: /forum/threadrate.php
Disallow: /forum/usercp.php
Disallow: /forum/usernote.php
Disallow: /forum/vbgooglemapme.php
Disallow: /forum/vBRadio.php
Disallow: /forum/vbseocp.php
Disallow: /roomeq/wizardinstallv3.29.zip
Disallow: /roomeq/wizardjarv3.29.zip
Disallow: /roomeq/oldindex.html

The Problem:

Google vacilates constantly between 6,000 and 30,000 entries. This board should have at least 300,000 or more entries.

So now the history:
This forum originally was on ubbthreads 6.5.1 at this domain: forums.robgalbraith.com. At this address and under this old domain name the board had over 400,000 entries. If you do a google search on this domain now it will still yield about 300,000 entries to the old board entires. As part of the deal the prior owner agreed to redirect traffic to our new url for 6 months. All fine and good, but I, in my ignorance did not negotiate in a (permanent) 301 redirect and the prior owner refused to reply to me at all when I made this request. The prior owner is now redirecting this domain to his main site at www.robgalbraith.com.

I guess my questions are:

1) how long will it take google to drop these old links now that they redirect to his main site?

2) how do I know if I am being penalized for perceived dup content at the old domain?

3) is this not the problem at all? The more I look at the facts the more it seems to be an issue of ubbthreads vs. vbulletin. Earlier, I left out the fact that I originally moved the forum to a new server on a different domain prophotocommunity (Pro Photo Home Forums). I left it on this domain for a few months while I tested vbulletin migration on our current domain and server. Well, during this time the plain vanilla Ubbthreads 6.5.2 had already received about 200,000 entries in google. I didn't even have a robots.txt file in place. If dup content were a problem then why in the world did google so readily index all of this same content at the new domain? The seo problems only started with the move to the new domain and vbulletin. I have implemented a 301 redirect from Pro Photo Home Forums and most of those old entries have been dropped for this domain.

All of this brings to mind the idea that I should just setup the current db on a new url using ubbthreads to see if google picks up the entries. There seems to be precious little to lose as we really have no pages indexed at this point and we have been at this new domain for about 4 months.

I am willing to engage some professional help, but am unsure as to whom or where I should go to solve this issue.

There is not much you can do about the old site redirects if you did not negotiate 301 redirects. Of course, you can continue to attempt to make the request to the owner.

Google fluctuates by nature. My primary suggestion would be to keep your vBulletin + vBSEO installation in place and unchanged. You need to establish some stability.

In the meantime, you can create backlinks and focus on building your content and community through general marketing and promotion.