Any of you guys have a site that Yahoo Slurp seems to love?
I've had problems with max database connections whenever Slurp decides to spend more than just a visit at my tntu.net site.
I'm using a "crawl delay" of 30 in my robots.txt file. I thought that would throttle the beast back a bit, but I don't think it has made much of a dent.
I currently have 48 users on my forums, 7 members, 10 visitors, 1 google adsense spider, and 30 Slurp agents.
In a day, I'll get around 900 visits from Slurp.
Anybody know a good way to calm that spider down?







Reply With Quote