vBulletin 4, the most powerful community software + vBSEO 3.5, the ultimate SEO solution = Your ultimate platform for 2010 and beyond. Click below to learn more.

View RSS Feed

Joe Ward

How much Googlebot time are you wasting?

Rate this Entry
by on 02-24-2009 at 06:13 PM (2535 Views)
Are you making the best use of Googlebot's time? If not, you are negatively impacting your SEO. One way to ensure that the time Googlebot spends on your forum is not wasted, is to get rid of inefficient, non-unique content. My focus here is on showpost pages.

While they are not "exact" copies of any other page on your site, all of the post content is already displayed on your thread pages. So they are partial duplicates.

If you want to find out how to remove them, check out this article:
http://www.vbseo.com/f2/how-replace-...iciency-31454/

Before you go ahead and do it, you should understand how much of Googlebot's time you are actually wasting. Let's do so first with an example.

Let's pretend you have a forum with:

  • 10,000 threads
  • 1,000,000 posts
  • 20 posts max per thread page
In this example, including multi-page threads, we could guestimate there would be about 50,000 thread pages in total. You can determine this exactly using your sitemap generation reports.


What percentage of total pages are actually thread pages with unique content?

  • 50,000 / (1,000,000 + 50,000) * 100 = 4.8% Unique Only!
  • In this example, the crawling efficiency is ONLY 4.8%. It is 95.2% INEFFICIENT!

How does that inefficiency delay SEO result for you?

  • Let's say Googlebot crawls 1,000 of your pages every single day.
  • If you enable showpost pages, how long would it take to complete?
1,050,000 pages / 1,000 Per Day = 1,050 DAYS... OR ALMOST 3 YEARS!
  • If you remove showpost pages, how long would it take to complete?
50,000 pages / 1,000 Per Day = 50 DAYS... OR JUST UNDER 2 MONTHS!
Now you can understand the impact of 95.2% inefficiency (in this example) can have on your SEO. Check out some other factors here.


Formula for Your Forum's Average Crawling Efficiency

Where:
  • P = Total Posts (Public)
  • T = Total Thread Pages (Including multi-page threads). Get from sitemap generator report.
Find out the total number of thread pages you have (including multi-page threads) from your sitemap generator report (T). Use the following formula:

Crawling Efficiency = T / (T + P) * 100
  • At vBSEO.com we use 15 posts per page.
  • Our sitemap generator reports 19,571 showthread pages total (public).
  • We'll guess that 60% of our 187,000 posts are public, so 112,200.
Our efficiency is 100%, because we replace showpost links with Permalinks.

However, if we did NOT do this, our efficiency would be approximately:

19,571 / ( 19,571 + 112,200) * 100 = 14.9% Crawling Efficiency OR 85.1% INEFFICIENCY.
If Googlebot hits only 1,000 of our pages a day:
  • We would be completely re-indexed with 20 DAYS since we have Permalinks enabled (replacing showpost).
  • If we kept the showpost pages in place, it could take 4 to 5 Months!
Categories
Uncategorized

Comments

  1. Shadab -
    Shadab's Avatar
    Nice article.

    Taking at look at my G Webmaster Tools stats,
    about 91% of the URLs in the sitemap are indexed.
    • |
    • permalink
  2. Future -
    Future's Avatar
    Why does googlebot keeps revisiting new threads couple of times everyday ?
    although those are not updated.
    • |
    • permalink
  3. Martyn -
    Martyn's Avatar
    Nice information here....
    • |
    • permalink

Trackbacks

Total Trackbacks 0
Trackback URL: