Page 1 of 2 1 2 LastLast
Results 1 to 15 of 19

Googlebot activity in the vbSEO log

This is a discussion on Googlebot activity in the vbSEO log within the Bug Reporting forums, part of the vBSEO SEO Plugin category; I'm seeing a discrepancy between my Who's Online and the vbSEO log for bots. I can see Googlebot hitting pages ...

  1. #1
    Senior Member
    Join Date
    Nov 2005
    Posts
    139
    Liked
    0 times

    Googlebot activity in the vbSEO log

    I'm seeing a discrepancy between my Who's Online and the vbSEO log for bots.

    I can see Googlebot hitting pages every minute or so throughout the day... appearing to use the same session throughout (unlike Yahoo it doesn't spawn tons of sessions).

    So I know and can observe that Googlebot is hitting showthread.php quite heavily.

    However when I look in my log:
    /vbseo_sitemap/index.php?hitdetails=20051201.log

    I'm seeing reports that Googlebot *NEVER* hit showthread, just forumdisplay.

    This is clearly incorrect, as I can observe the bots but the stats never reflect what I observe. How are you identifying Googlebots?

    As an additional question... on top of the above I also see Google AdSense spiders with the user-agent "Mediapartners-Google/2.1", are you also including the Google AdSense spider in the Google totals? If not, could you add them to the bot list (unsure how you would do this, as your code is encoded).

  2. #2
    vBSEO Staff Oleg Ignatiuk's Avatar
    Real Name
    Oleg Ignatiuk
    Join Date
    Jun 2005
    Location
    Belarus
    Posts
    25,689
    Liked
    157 times
    Hello,

    hm. we never had this problem - showthread pages are shown in SE Bots Activity Log (as well as other pages). Do you have this problem for a long time or is it just a one day logged issue?
    PM sent.

    How are you identifying Googlebots?
    It is identified by a "googlebot" substring in User-agent header.

    As an additional question... on top of the above I also see Google AdSense spiders with the user-agent "Mediapartners-Google/2.1",
    Sure, it will be added in the next vBSEO release.
    Oleg Ignatiuk / Crawlability Inc.
    vBSEO 3.6.0 GOLD Released!
    Unveiling the NEW vBSEO Sitemap Generator 3.0. - available NOW for vBSEO Customers!


  3. #3
    Senior Member Thomas Hoehl's Avatar
    Real Name
    Thomas
    Join Date
    Jul 2005
    Location
    Ockershausen, Germany
    Posts
    1,329
    Liked
    3 times
    Quote Originally Posted by Oleg Ignatiuk

    Sure, it will be added in the next vBSEO release.
    I think, that is not so good, because this bot is only for Adsense and when this bot crawl your Site, no Link of them is come in Google-Index. He only look for relevant Ads to show.

  4. #4
    vBSEO Staff Oleg Ignatiuk's Avatar
    Real Name
    Oleg Ignatiuk
    Join Date
    Jun 2005
    Location
    Belarus
    Posts
    25,689
    Liked
    157 times
    It is supposed to be in a separate column and doesn't mixed with googlebot stats!
    Oleg Ignatiuk / Crawlability Inc.
    vBSEO 3.6.0 GOLD Released!
    Unveiling the NEW vBSEO Sitemap Generator 3.0. - available NOW for vBSEO Customers!


  5. #5
    Senior Member Thomas Hoehl's Avatar
    Real Name
    Thomas
    Join Date
    Jul 2005
    Location
    Ockershausen, Germany
    Posts
    1,329
    Liked
    3 times
    That is o.K.

  6. #6
    Senior Member
    Join Date
    Nov 2005
    Posts
    139
    Liked
    0 times
    Quote Originally Posted by Oleg Ignatiuk
    Hello,

    hm. we never had this problem - showthread pages are shown in Search Engine Bots Activity Log (as well as other pages). Do you have this problem for a long time or is it just a one day logged issue?
    PM sent.
    I'm not sure when it started... I've attached an image showing that yes, show Googlebot's are hitting showthread.php as you can see by the last row.

    You still have the password I gave you for my stats/log files, and you'll see in there that the value on showthread.php for Googlebot is still zero.

    I took the screenshot with the user agents in if it helps, not sure if it will.

    As I've remarked elsewhere, I would usually track down and debug a bug like this myself, but your encoded code makes this difficult to do (not worth the effort), so I'll just help how I can and leave the work to you
    Attached Thumbnails Attached Thumbnails untitled-1.gif  

  7. #7
    Senior Member
    Join Date
    Nov 2005
    Posts
    139
    Liked
    0 times
    OK, you can ignore me now. It finally says 1!

  8. #8
    vBSEO Staff Oleg Ignatiuk's Avatar
    Real Name
    Oleg Ignatiuk
    Join Date
    Jun 2005
    Location
    Belarus
    Posts
    25,689
    Liked
    157 times
    It also shows 1 for 2005-12-01 and 6 for 2005-11-28!

    Let's see how it will go further.

    Do you have some site statistics analyzer software installed to get googlebot hits numbers and compare it to the numbers reported by SE Bots Activity Log? (are they in nearly the same range?)
    Oleg Ignatiuk / Crawlability Inc.
    vBSEO 3.6.0 GOLD Released!
    Unveiling the NEW vBSEO Sitemap Generator 3.0. - available NOW for vBSEO Customers!


  9. #9
    Senior Member
    Join Date
    Nov 2005
    Posts
    139
    Liked
    0 times
    No, I've turned off all logging. I'm a performance freak and Google Analytics already do all my stats (but not bots unfortunately).

    I might add something to copy sessions into a second table where the googlebot useragent is detected... that would give me a pretty complete log of it. I'll think about doing that tomorrow if I haven't seen the number rise inline with the spiders I'm seeing through Who's Online.

  10. #10
    Senior Member
    Join Date
    Nov 2005
    Posts
    139
    Liked
    0 times
    OK, I have about 30 Googlebots on my site, nearly all of them looking at threads. Yet the stats show nothing.

    So my next question: Are the stats updated instantly? Or do they accumulate elsewhere (where) only to be updated hourly (or something)?

    If they're instant, then there is a problem, and I'll go and code a second session table to keep track of all bots for you to use for debugging.

  11. #11
    vBSEO Staff Oleg Ignatiuk's Avatar
    Real Name
    Oleg Ignatiuk
    Join Date
    Jun 2005
    Location
    Belarus
    Posts
    25,689
    Liked
    157 times
    Yes, results are updated instantly (in real-time). And if you can prepare a session table with date/time, page url, bot user-agent that would be helpful for troublehooting, thanks!
    Oleg Ignatiuk / Crawlability Inc.
    vBSEO 3.6.0 GOLD Released!
    Unveiling the NEW vBSEO Sitemap Generator 3.0. - available NOW for vBSEO Customers!


  12. #12
    Senior Member
    Join Date
    Nov 2005
    Posts
    139
    Liked
    0 times
    I've had a look at the class_core.php file of vBulletin where the session stuff is, and can't find a trivial way of adding what I would want so that I can log just Googlebot visits.

    I've also thought about it and come to the conclusion that it doesn't matter much Yes my Googlebot visits according to Who's Online greatly outnumber the stats in the vbseo stats page... but on reflection what matters is that Googlebot is visiting me in good numbers.

    So I'm actually going to drop this in priority. I didn't have stats on bots before, so I'm not going to miss them much

    Thanks anyway, I apologise for not being able to figure out class_core.php so we could resolve the bug.

  13. #13
    vBSEO Staff Oleg Ignatiuk's Avatar
    Real Name
    Oleg Ignatiuk
    Join Date
    Jun 2005
    Location
    Belarus
    Posts
    25,689
    Liked
    157 times
    Sure, I see. Hopefully we'll be able to find this issue somehow
    Oleg Ignatiuk / Crawlability Inc.
    vBSEO 3.6.0 GOLD Released!
    Unveiling the NEW vBSEO Sitemap Generator 3.0. - available NOW for vBSEO Customers!


  14. #14
    Senior Member
    Real Name
    TopAs
    Join Date
    Jul 2005
    Location
    Isernhagen near Hannover Germany
    Posts
    171
    Liked
    0 times
    The Hits are not counted in my Bots activity log - what am I doing wrong ?
    I set this option to yes : Log Spider Activity in your Forums?

    I am using VBSEO 2.3.2 and 1.5 Sitemap

  15. #15
    vBSEO Staff Oleg Ignatiuk's Avatar
    Real Name
    Oleg Ignatiuk
    Join Date
    Jun 2005
    Location
    Belarus
    Posts
    25,689
    Liked
    157 times
    Please check if the "vbseo_sitemap/data/hits/" folder exists at your host (and if not, try to create it manually and set 0666 permissions to it).
    Oleg Ignatiuk / Crawlability Inc.
    vBSEO 3.6.0 GOLD Released!
    Unveiling the NEW vBSEO Sitemap Generator 3.0. - available NOW for vBSEO Customers!


Page 1 of 2 1 2 LastLast

Similar Threads

  1. vBSEO 2.4.0 Released - Includes Google AdSense Targeting Feature!
    By Juan Muriente in forum vBSEO Announcements
    Replies: 74
    Last Post: 05-20-2006, 10:29 PM
  2. vBSEO 2.0 RC7 Released
    By Juan Muriente in forum vBSEO Announcements
    Replies: 17
    Last Post: 09-09-2005, 12:00 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •