SEO

Any of you guys have a site that Yahoo Slurp seems to love?

I've had problems with max database connections whenever Slurp decides to spend more than just a visit at my tntu.net site.

I'm using a "crawl delay" of 30 in my robots.txt file. I thought that would throttle the beast back a bit, but I don't think it has made much of a dent.

I currently have 48 users on my forums, 7 members, 10 visitors, 1 google adsense spider, and 30 Slurp agents.

In a day, I'll get around 900 visits from Slurp.

Anybody know a good way to calm that spider down?

I guess a few people have this problem, I'll just keep bumping up that crawl limit for slurp and see if that works...
yahoo slurp

i actually got a response from Yahoo the other day, they suggest to increase the crawl delay to 120 or 240 to alleviate the problem, it seems to have worked, hopefully it doesn't affect the Yahoo indexing, as slow as it is already.

Wow - great that they got back to you about that. Interesting.

__________________
Joe Ward / Crawlability Inc.
Support Team Launches New DeskPro Powered Tool Enhanced Support at Your Service

vBSEO 3.2.0 Launched - Maximum Overdrive for Your Web Traffic! Over 100 Instant SEO Optimizations

6X Traffic - $1400 in One Day with vBSEO! Imagine What the vBSEO Patent Pending Technology Can Do For You.

Don't be alarmed by the sheer number of Yahoo Slurps,the reason there are lots of them at the same time is because they came from different IPs.
Like right now I have one Googlebot and 30 Slurps and it’s almost always like that. But at the end of the day, hits from Googlebot far exceed Yahoo Slurp.

In this very moment, I have the following bots in my "who's online"

Google (3), Google AdSense (2), Yahoo! Slurp (264), MSNBot (4)




I've seen more than 400 on some days.

same here. my current at 2:30 am:

Yahoo! Slurp (329), Google (2), AskJeeves (1), Google AdSense (1), MSNBot (1)





I had to modify my maxclients setting because of it.


[/usr/local/apache/conf/httpd.conf]
KeepAlive On
MaxKeepAliveRequests 1000
KeepAliveTimeout 2
MinSpareServers 10
MaxSpareServers 15
StartServers 10
MaxClients 200
MaxRequestsPerChild 1000

There are days that I really hate Yahoo.

Ok this is getting out of control:

Yahoo! Slurp (376), Google AdSense (2), Google (1), AskJeeves (1), MSNBot (1)


i'm with ya guys... lol

Yahoo! Slurp (512), Google (2), AskJeeves (1), Google AdSense (2)

lol

update:

Yahoo! Slurp (886), Google (2), MSNBot (3), Google AdSense (1)

I've since set my Slurp crawl delay to 60 and now I'm averaging 50-75 of the buggers at any time. Before that it was 300-500.

is slurp delay defined in robots.txt?

If so, doesn't it affect indexing speed?

yahoo indexing was notoriously slow for me anyway, I haven't noticed a difference, but maybe that's because I'm submitting a yahoo sitemap too.

the problem that I had with Yahoo is that sometimes it'd spike and use up all my mysql connections. My changes above helped my bandwidth resources a little bit too.