200 bots on my site for the last 2days, killin my load

My server is not handling all these bots too well, should I expect this will be the norm to have this many? Currently i have about 180 yahoo slurps, 5 googlebot, 4 msnbot, and a few others. I put a crawl delay of 10 for yahoo but not sure it's being followed. Typically my site has 200 members 50 guests online at peak with a load of 1.5, post vbseo and opening things up more to bots (i used to block most bots) my load is fluctuating from 8.0 to 11.0 to 33.0 etc. The result is a slow site but if this 'storm' will pass then i'm ok with that, if not then i guess i may need to upgrade my server, but i currently have dual proc and that's hitting 33.0 so lord knows what kind of box i would need. I' just hoping this passes. Is there no way to reduce how many slurps spiders i have? it seems other bots keep themselves in check but here is yahoo with 180 of them on my site! ugh..

ok i banned msnbot and my load has dropped back to 2.5 and holding. anyone else have problems with msnbot loading your box or maybe it's just a coincidence? hmm

Sross,

I am sorry to hear about that many bots ... How i wish they could visit my forum

What we have here is to a setup a WebServer Farm and load balance them using a pair of Load Balancer.

So far it has been working well.

Cheers,
Edy

get a bigger hosting package if you can't handle the bots

you rarely want to ban bots..

well currently i have a dual opteron 246, 4gigs ram, 72gb scsi. my load was usually around 1-1.5 with 200-250 or so users during peak. I don't know if that's good or not it's hard to find comparisons. I am debating upgrading the cpu's..

I would think 4 gigs could handle it, ya I might think about upgrading the cpu, Fluid Hosting: FreeBSD and Windows Hosting is a pretty good host if your looking