sitemap URLs vs indexed URLs in Google for a new site

We just launched a new vBulletin forum running vbSEO. The forum contains >200,000 messages imported from an old non-vBulletin forum.

After one week online I can see when I check the sitemap page at Google that our sitemaps contain 65,000 URLs and that about 4,000 URLs have been indexed so far. Is this normal? Can I expect the "indexed so far" number to continue to increase and come close to the 65,000 URLs our sitemaps contain?

If I do a few test-searches on Google I can find messages from the first few pages in all forums, but old threads are not available in Google yet.

Should I just wait for Google to continue indexing our site or does the "4,000 indexed URL" / "65,000 sitemap URLs" ratio after one week indicate a problem which I should be searching for?

Google currently downloads about 6,000 threads per day from our site. I don't know if this is normal or not. The number of indexed URLs is not growing at all that quick though. I assume that most of the downloads from googlebot are re-downloads of already indexed threads.

It will never be 100%. 98% might be pushing it.

Keep in mind, the sitemap is merely a 'suggested list of urls to index'.

In time, it will increase.

Thanks for your reply briansol. Your guide was an important help in setting up vbSEO . I read between your lines that for a good forum design using vbSEO set up correctly I can expect to eventually get most of my sitemap URLs indexed by Google.

Have you ever re-organized a large forum and changed all URLs and seen how long it takes for the re-organized forum to get re-indexed by Google? How large percentage of the sitemap URLs do eventually get indexed in Google for your forums?

I would like to know if I should look for problems or just sit back and wait.

i did that exact thing with my own site.

i was on ipb until 2006 when i switched to vb BECAUSE of vbseo. i had roughly 450k posts at the time.
if you followed my guide, i wouldn't touch anything for at least a week. google is NOT instant by any means.

Almost a week has passed since my first post in this thread. The number of indexed URLs shown in the sitemap section of my google webmaster tools has now increased to 13,000. Better, but still far from the 63,000 URLs my sitemap contains, but it is still increasing at least. I wonder how long it will take for Google to index most of the URLs.

Now up at 16,000 indexed URLs.... this indexing is going nervously slow. I hope that there are no problems with the vbSEO setup that causes this slowness. Anyone have any experience on how long it would normally take for Google to index a site with something like 60,000 URLs?

it will NEVER be 100%.

i've had this on 3 years, and i'm still not 100%

Sitemap stats
Total URLs: 119308
Indexed URLs: 95281

Yea, I'm not dreaming of 100%, but we are just at 25% now after more than two weeks online and with about 6,000 googlebot downloads per day. Do you have any memory of how long time it took for your site to get above say 80000 indexed URLs, or above 70% ?

Well, the number of urls in my sitemap goes up on a daily basis. It will CONSTANTLY play catch up, until you close your forum for posting.

Yep, have you got any idea how long it took to get above ~70% indexed URLs after you launched your new vBulletin+vbSEO forum? Am I just too anxious and afraid of problems and should I just sit back and wait for a few months and not worry about being down below 50% indexed URLs for a month or two?

I don't recall exact times, but it wasn't overnight. As long as you're seeing it increase, i wouldn't worry about it.

Thanks for your advice. I'm sorry to keep nagging about these indexed URLs... they are indeed growing steadily and I will relax for a month or two before I start searching for a bug in my setup that might not exist. Thanks briansol!

Do you include posts in your sitemap? Or is it shut off. Some say it's duplicate content.

Individual posts are not included in the sitemap and are also turned off in the forums just to avoid that duplicate content issue.

The number of indexed URLs is still slowly increasing and we are now up at 24,000 indexed URLs out of a total of 65,000 URLs in the sitemap.

i suggest using the "replace showpost with permalink" option, robots.txt out showpost.php, and not inclduing them in the sitemap either.