Sitemaps

I installed VBSeo Sitemap Generator recently.. never had got round to doing a sitemap for my forum before. Ive been checking how it would perform getting the older stuff indexed and it hasnt really increased to much so far, I was just checking stuff in Google webmaster tools and found the following statistics:

Sitemap statistics:
Total URLs: 2024756
Indexed URLs: 32586


Is that a usual ratio, will Google eventually index all the urls?

It will grow over time. It varies from forum to forum. The higher the # of the posts (millions+) the less likely it is that full indexing will be achieved.

However, as further authority with the search engines is achieved, and more links created, the deeper you can expect it to crawl into your site.

I thought I should update.. approx 6 weeks later?

Total URLs: 2024756
Indexed URLs: 139029

Looking a lot nicer, strangely the total hasnt changed but the indexed is going up

Hello Chris ,

It looks like you added showpost urls to your sitemap files. I strongly advise you not to include showpost urls to sitemap.

why?

Originally Posted by plusEV

why?

Duplicate content.

The content in the individual showpost pages is already included in the showthread pages.

Originally Posted by Shadab

Duplicate content.

The content in the individual showpost pages is already included in the showthread pages.

ok. ty 4 ur reply.

Ok, we should not include showpost threads, but WHY you added this option then?

Because it is an option. Some people want to index it some people dont.

its an option to make DC, thats no real option Oo

Except for a thread that contains only 1 post, showthread and showpost pages are not *exactly* duplicate content.

More specifically, a showpost page contains a *portion* of content that is also included within the *full content* of a show post page.

To maintain full compatibility with vBulletin, which includes showpost pages, we must include this option.

However, my recommendation would be to not use showpost pages as they greatly diminish crawling efficiency, without adding any *new content*, even if they are not exact duplicate copies of the showthread pages.

Compare the # of threads to the # of total posts on any forum, and you will quickly see that including the showpost pages will dramatically increase the indexing workload of the search engines.

When the SE bots visit a forum and start indexing 1000s of pages, I would certainly prefer that all these pages be thread pages with unique content, and not extra showpost pages whose content is already included elsewhere.

For those who prefer including showpost, our sitemap also accommodates that option.