Googlebot activity in the vbSEO log

I'm seeing a discrepancy between my Who's Online and the vbSEO log for bots.

I can see Googlebot hitting pages every minute or so throughout the day... appearing to use the same session throughout (unlike Yahoo it doesn't spawn tons of sessions).

So I know and can observe that Googlebot is hitting showthread.php quite heavily.

However when I look in my log:
/vbseo_sitemap/index.php?hitdetails=20051201.log

I'm seeing reports that Googlebot *NEVER* hit showthread, just forumdisplay.

This is clearly incorrect, as I can observe the bots but the stats never reflect what I observe. How are you identifying Googlebots?

As an additional question... on top of the above I also see Google AdSense spiders with the user-agent "Mediapartners-Google/2.1", are you also including the Google AdSense spider in the Google totals? If not, could you add them to the bot list (unsure how you would do this, as your code is encoded).

Hello,

hm. we never had this problem - showthread pages are shown in SE Bots Activity Log (as well as other pages). Do you have this problem for a long time or is it just a one day logged issue?
PM sent.

How are you identifying Googlebots?

It is identified by a "googlebot" substring in User-agent header.

As an additional question... on top of the above I also see Google AdSense spiders with the user-agent "Mediapartners-Google/2.1",

Sure, it will be added in the next vBSEO release.

Originally Posted by Oleg Ignatiuk

Sure, it will be added in the next vBSEO release.

I think, that is not so good, because this bot is only for Adsense and when this bot crawl your Site, no Link of them is come in Google-Index. He only look for relevant Ads to show.

It is supposed to be in a separate column and doesn't mixed with googlebot stats!

That is o.K.

Originally Posted by Oleg Ignatiuk

Hello,

hm. we never had this problem - showthread pages are shown in Search Engine Bots Activity Log (as well as other pages). Do you have this problem for a long time or is it just a one day logged issue?
PM sent.

I'm not sure when it started... I've attached an image showing that yes, show Googlebot's are hitting showthread.php as you can see by the last row.

You still have the password I gave you for my stats/log files, and you'll see in there that the value on showthread.php for Googlebot is still zero.

I took the screenshot with the user agents in if it helps, not sure if it will.

As I've remarked elsewhere, I would usually track down and debug a bug like this myself, but your encoded code makes this difficult to do (not worth the effort), so I'll just help how I can and leave the work to you

OK, you can ignore me now. It finally says 1!

It also shows 1 for 2005-12-01 and 6 for 2005-11-28!

Let's see how it will go further.

Do you have some site statistics analyzer software installed to get googlebot hits numbers and compare it to the numbers reported by SE Bots Activity Log? (are they in nearly the same range?)

No, I've turned off all logging. I'm a performance freak and Google Analytics already do all my stats (but not bots unfortunately).

I might add something to copy sessions into a second table where the googlebot useragent is detected... that would give me a pretty complete log of it. I'll think about doing that tomorrow if I haven't seen the number rise inline with the spiders I'm seeing through Who's Online.

OK, I have about 30 Googlebots on my site, nearly all of them looking at threads. Yet the stats show nothing.

So my next question: Are the stats updated instantly? Or do they accumulate elsewhere (where) only to be updated hourly (or something)?

If they're instant, then there is a problem, and I'll go and code a second session table to keep track of all bots for you to use for debugging.

Yes, results are updated instantly (in real-time). And if you can prepare a session table with date/time, page url, bot user-agent that would be helpful for troublehooting, thanks!

I've had a look at the class_core.php file of vBulletin where the session stuff is, and can't find a trivial way of adding what I would want so that I can log just Googlebot visits.

I've also thought about it and come to the conclusion that it doesn't matter much Yes my Googlebot visits according to Who's Online greatly outnumber the stats in the vbseo stats page... but on reflection what matters is that Googlebot is visiting me in good numbers.

So I'm actually going to drop this in priority. I didn't have stats on bots before, so I'm not going to miss them much

Thanks anyway, I apologise for not being able to figure out class_core.php so we could resolve the bug.

Sure, I see. Hopefully we'll be able to find this issue somehow

The Hits are not counted in my Bots activity log - what am I doing wrong ?
I set this option to yes : Log Spider Activity in your Forums?

I am using VBSEO 2.3.2 and 1.5 Sitemap

Please check if the "vbseo_sitemap/data/hits/" folder exists at your host (and if not, try to create it manually and set 0666 permissions to it).