More threads in sitemap than actually in forum?

[GBSF]

How is this possible? My forum says it has 22,514 threads and I just generated a sitemap and it says the following:

Forum Display: 1076
Show Thread: 69810
Show Post:
Member Profiles:
Poll Results: 330
Archive: 0

Total Indexed URLs: 71469
Total Processing Time: 74.39 seconds

Google ping: Successful.
Yahoo ping: Successful.
Ask ping: Successful.
Moreover ping: Successful.
Live.com ping: Successful.

[Mert Gökçeimam]

Hello Joey ,

Sitemap Generator also includes pagination for threads.

[Joe Ward]

FYI - Knowing the total # of pages (including pagination) is very useful. Check out:
How much Googlebot time are you wasting? - vBulletin SEO Forums

[GBSF]

FYI - Knowing the total # of pages (including pagination) is very useful. Check out:
How much Googlebot time are you wasting? - vBulletin SEO Forums

Hey Joe,

Just read through your tutorial, great work!

I followed your advice and replaced showpost links with permalinks. However I am a little confused now. If I did this correctly, my crawling efficiency actually went down? Here is what I have, maybe you could enlighten me a bit:

I have a total of 232102 posts on the forum.

Here is my sitemap report before I did the show post links with permalinks change:

Forum Display: 1076
Show Thread: 69810
Show Post:
Member Profiles:
Poll Results: 330
Archive: 0

Total Indexed URLs: 71469
Total Processing Time: 74.39 seconds


Based on these stats, I calculated my crawling efficiency to be about 23 percent.

Here is my sitemap report after I made the change to show post with permalinks:

Forum Display: 810
Show Thread: 63300
Show Post:
Member Profiles:
Poll Results: 297
Archive: 0

Total Indexed URLs: 64671
Total Processing Time: 56.70 seconds


Based on the new numbers, my crawling efficiency is now 21 percent. How could be? What am I missing? It seems to have gotten worse instead of better.

[Joe Ward]

Formula for Your Forum's Average Crawling Efficiency

Where:

• P = Total Posts (Public)
• T = Total Thread Pages (Including multi-page threads). Get from sitemap generator report.

Find out the total number of thread pages you have (including multi-page threads) from your sitemap generator report (T). Use the following formula:

Crawling Efficiency = T / (T + P) * 100

Ignoring other page types, your crawling efficiency is now 100%!

The forum above is *only* for calculating efficiency when you have *showposts enabled*.

Once you turn them off, P is removed from equation (P = 0), and it becomes:

T / ( T + 0) * 100

or

T / T * 100 = 100%


Did you check to see if the showpost pages were generating any traffic first?
Do your ShowPost Pages Get Search Hits? - vBulletin SEO Forums

[GBSF]

Thanks for the quick reply Joe!

So with what you mentioned, then since my show posts were already turned off in the sitemap generator prior, then my crawling efficiency was 100% as well before I even did the show post with permalinks change, correct?

I didn't check to see if those showpost pages were generating traffic yet, but I have had that setting turned off in the sitemap generator for some time now.

By the way, I just upgraded the generator to 2.5 from 2.2.

[Joe Ward]

Actually - no, you cannot get the 100% efficiency unless you replace showpost links with Permalinks, because the search engines can still find them and crawl them.

The sitemaps do tell the engines what you have available to be crawled, but they do not restrict it from crawling anything else they find on your site.

[GBSF]

OK makes sense....

By the way, I check checked in my google analytics and for the last month, I have had only 11 visits to landing pages that contain "-post".

I guess i'm ok then with leaving show posts off in sitemap generator?

[Joe Ward]

Yes, I would say 100% crawling efficiency is a great trade for 11 visits.

[GBSF]

Very cool! Thank you for all your help! I greatly appreciate it!

[GBSF]

Joe, 2 more questions while I am in the SEO mine set...

Should I turn off Archive or leave it on in VB? I have VBSEO set to point the links in the archives to the actual threads currently.


How often should I have my sitemap generate automatically? Every night and ping search engines ok?

Thanks again!

[briansol]

I suggest disabling the archive entirely. with vbseo, its un-necessary.

sitemap should run once a day at your lowest server use time (4am ish)