bots getting busy
Posted 01-20-2008 at 12:21 AM by JADM
Well, traffic was off a little today, maybe because of my lost day yesterday (uniques were about the same percentage, though)...
Only real news is that with 45 min. left in the day I got 337 bots today. This is more than double the average number of bots. They were looking at showthread (155) and member profile (151). I changed my user titles from "New Member" to "New PEB Forum Member" today for users with no posts. Since "PEB" is one of my keywords, hopefully this will have some effect on SERP.
My indexed pages haven't really jumped up (938 on google). Again, hopefully these bots going crazy today will help.
Only real news is that with 45 min. left in the day I got 337 bots today. This is more than double the average number of bots. They were looking at showthread (155) and member profile (151). I changed my user titles from "New Member" to "New PEB Forum Member" today for users with no posts. Since "PEB" is one of my keywords, hopefully this will have some effect on SERP.
My indexed pages haven't really jumped up (938 on google). Again, hopefully these bots going crazy today will help.
Total Comments 7
Comments
-
Awesome!
Hey, does the MSN bot seem to be the slowest to kick in for you too? I always notice Google seems to hit the majority of my content first, then Yahoo, and like a week later MSN finally bombards my site.Posted 01-20-2008 at 07:20 PM by Citricguy
-
MSN crawls your site!?!?!?
Joking, but what is really going on is msn hit me on the third and fourth day with 98 and 93 crawls...since then only 2 or 3 times a day. So, I am waiting for them to catch up, I guess. I have only been running both products for 20 days, so it is early.
I really wish I understood more about the Sitemap generator. I suspect part is that with the algorithms and such being closely held secrets and things changing, it may be hard to figure this stuff out. But from the vBSEO side, I wish they would explain the products typical performane a little better. I can only figure they are busy with other stuff or else don't want to give out too much info for fear of getting their program ripped off. For me, knowing and understanding things a little more would help me in figuring out if my results are as to be expected. My big fear, after reading different reviews on the forums is that I end being one of those guys who wait 7 months and then is like, "hey, this isn't working!" and come to find out I am doing something dumb. Still, I am happy / confident things will go well. Except for today with my traffic off by 2/3 (I am blaming football games).Posted 01-20-2008 at 09:20 PM by JADM - where are you getting all these bot stats from?Posted 01-21-2008 at 04:37 AM by hornstar6969
-
Admincp
Goto AdminCP >> VBSEO >> VBSEO Sitemap Reports >> Show SE Bots Activity Log .
That's how I do it, though, you could go to server logs and see, too (I guess...but I think it is not broken down as well, segregated from the other visitors).Posted 01-21-2008 at 10:16 AM by JADM - I'm going to go ahead and give vBSEO and vBSiteMap Generator credit for getting pages indexed within 1-4 hours upon creation (with a little help from feedburner.)
Yay PingShot
All my tags were indexed within 24 hours too when i upgraded to 3.7.Posted 01-21-2008 at 05:13 PM by Citricguy -
Since your crawling was about 50% member profiles vs. threads, that is one of the reasons I (personally) prefer blogging those member profile pages. I'd rather have the bots prioritize on the threads.Posted 01-22-2008 at 10:42 PM by Joe Ward 
-
Do the bots act that way?
Joe,
I have been trying to figure that out. If there are a 100 bots crawling your site on a given day, and 50% crawl profile, if you had it to do over again and you block profiles, would the same 100 bots come to your site? Or would the SE say, "hey, there is no profile to crawl, let's only send 50 bots"?Posted 01-25-2008 at 01:37 AM by JADM
Total Trackbacks 0
