Enhancing 80 million pages.

Register FAQ Members List Social Groups Calendar Search Today's Posts Mark Forums Read

Rate this Entry

bots getting busy

Posted 01-20-2008 at 12:21 AM by JADM
Well, traffic was off a little today, maybe because of my lost day yesterday (uniques were about the same percentage, though)...

Only real news is that with 45 min. left in the day I got 337 bots today. This is more than double the average number of bots. They were looking at showthread (155) and member profile (151). I changed my user titles from "New Member" to "New PEB Forum Member" today for users with no posts. Since "PEB" is one of my keywords, hopefully this will have some effect on SERP.

My indexed pages haven't really jumped up (938 on google). Again, hopefully these bots going crazy today will help.

Posted in Uncategorized
Views 669 Comments 7 Email Blog Entry
Total Comments 7

Comments

  1. Old
    Citricguy's Avatar
    Awesome!

    Hey, does the MSN bot seem to be the slowest to kick in for you too? I always notice Google seems to hit the majority of my content first, then Yahoo, and like a week later MSN finally bombards my site.
    permalink
    Posted 01-20-2008 at 07:20 PM by Citricguy Citricguy is offline
  2. Old
    JADM's Avatar

    MSN crawls your site!?!?!?

    Joking, but what is really going on is msn hit me on the third and fourth day with 98 and 93 crawls...since then only 2 or 3 times a day. So, I am waiting for them to catch up, I guess. I have only been running both products for 20 days, so it is early.

    I really wish I understood more about the Sitemap generator. I suspect part is that with the algorithms and such being closely held secrets and things changing, it may be hard to figure this stuff out. But from the vBSEO side, I wish they would explain the products typical performane a little better. I can only figure they are busy with other stuff or else don't want to give out too much info for fear of getting their program ripped off. For me, knowing and understanding things a little more would help me in figuring out if my results are as to be expected. My big fear, after reading different reviews on the forums is that I end being one of those guys who wait 7 months and then is like, "hey, this isn't working!" and come to find out I am doing something dumb. Still, I am happy / confident things will go well. Except for today with my traffic off by 2/3 (I am blaming football games).
    permalink
    Posted 01-20-2008 at 09:20 PM by JADM JADM is offline
  3. Old
    where are you getting all these bot stats from?
    permalink
    Posted 01-21-2008 at 04:37 AM by hornstar6969 hornstar6969 is offline
  4. Old
    JADM's Avatar

    Admincp

    Goto AdminCP >> VBSEO >> VBSEO Sitemap Reports >> Show SE Bots Activity Log .

    That's how I do it, though, you could go to server logs and see, too (I guess...but I think it is not broken down as well, segregated from the other visitors).
    permalink
    Posted 01-21-2008 at 10:16 AM by JADM JADM is offline
  5. Old
    Citricguy's Avatar
    I'm going to go ahead and give vBSEO and vBSiteMap Generator credit for getting pages indexed within 1-4 hours upon creation (with a little help from feedburner.) Yay PingShot

    All my tags were indexed within 24 hours too when i upgraded to 3.7.
    permalink
    Posted 01-21-2008 at 05:13 PM by Citricguy Citricguy is offline
  6. Old
    Joe Ward's Avatar
    Since your crawling was about 50% member profiles vs. threads, that is one of the reasons I (personally) prefer blogging those member profile pages. I'd rather have the bots prioritize on the threads.
    permalink
    Posted 01-22-2008 at 10:42 PM by Joe Ward Joe Ward is online now
  7. Old
    JADM's Avatar

    Do the bots act that way?

    Joe,

    I have been trying to figure that out. If there are a 100 bots crawling your site on a given day, and 50% crawl profile, if you had it to do over again and you block profiles, would the same 100 bots come to your site? Or would the SE say, "hey, there is no profile to crawl, let's only send 50 bots"?
    permalink
    Posted 01-25-2008 at 01:37 AM by JADM JADM is offline
 
Total Trackbacks 0

Trackbacks


All times are GMT -4. The time now is 05:54 PM.


Powered by vBulletin Version 3.8.0 Release Candidate 1
Copyright ©2000 - 2008, Jelsoft Enterprises Ltd.
SEO by vBSEO 3.2.5 ©2008, Crawlability, Inc.