vBulletin Search Engine Optimization
This is a discussion on Good problem to have with yahoo slurp within the Analysis: Traffic & SERPS forums, part of the vBulletin SEO Discussion category; Any of you guys have a site that Yahoo Slurp seems to love? I've had problems with max database connections ...
| |||||||
Enhancing 80 million pages. | Register | FAQ | Members List | Social Groups | Calendar | Search | Today's Posts | Mark Forums Read |
|
#1
| |||
| |||
| Good problem to have with yahoo slurp
Any of you guys have a site that Yahoo Slurp seems to love? I've had problems with max database connections whenever Slurp decides to spend more than just a visit at my tntu.net site. I'm using a "crawl delay" of 30 in my robots.txt file. I thought that would throttle the beast back a bit, but I don't think it has made much of a dent. I currently have 48 users on my forums, 7 members, 10 visitors, 1 google adsense spider, and 30 Slurp agents. In a day, I'll get around 900 visits from Slurp. Anybody know a good way to calm that spider down? |
|
#2
| |||
| |||
|
I guess a few people have this problem, I'll just keep bumping up that crawl limit for slurp and see if that works... yahoo slurp |
|
#3
| |||
| |||
|
i actually got a response from Yahoo the other day, they suggest to increase the crawl delay to 120 or 240 to alleviate the problem, it seems to have worked, hopefully it doesn't affect the Yahoo indexing, as slow as it is already.
|
|
#4
| ||||
| ||||
|
Wow - great that they got back to you about that. Interesting.
__________________ Joe Ward / Crawlability Inc. Support Team Launches New DeskPro Powered Tool Enhanced Support at Your Service vBSEO 3.2.0 Launched - Maximum Overdrive for Your Web Traffic! Over 100 Instant SEO Optimizations 6X Traffic - $1400 in One Day with vBSEO! Imagine What the vBSEO Patent Pending Technology Can Do For You. |
|
#5
| |||
| |||
|
Don't be alarmed by the sheer number of Yahoo Slurps,the reason there are lots of them at the same time is because they came from different IPs. Like right now I have one Googlebot and 30 Slurps and it’s almost always like that. But at the end of the day, hits from Googlebot far exceed Yahoo Slurp. |
|
#6
| ||||
| ||||
|
In this very moment, I have the following bots in my "who's online" Google (3), Google AdSense (2), Yahoo! Slurp (264), MSNBot (4) I've seen more than 400 on some days. |
|
#7
| ||||
| ||||
|
same here. my current at 2:30 am: Yahoo! Slurp (329), Google (2), AskJeeves (1), Google AdSense (1), MSNBot (1) I had to modify my maxclients setting because of it. [/usr/local/apache/conf/httpd.conf] KeepAlive On MaxKeepAliveRequests 1000 KeepAliveTimeout 2 MinSpareServers 10 MaxSpareServers 15 StartServers 10 MaxClients 200 MaxRequestsPerChild 1000 |
|
#8
| ||||
| ||||
|
There are days that I really hate Yahoo. Quote:
|
|
#9
| ||||
| ||||
|
Ok this is getting out of control: Quote:
|
|
#10
| ||||
| ||||
| Yahoo! Slurp (376), Google AdSense (2), Google (1), AskJeeves (1), MSNBot (1) i'm with ya guys... lol |
|
#11
| ||||
| ||||
| Yahoo! Slurp (512), Google (2), AskJeeves (1), Google AdSense (2) lol |
|
#12
| ||||
| ||||
|
update: Yahoo! Slurp (886), Google (2), MSNBot (3), Google AdSense (1) |
|
#13
| ||||
| ||||
|
I've since set my Slurp crawl delay to 60 and now I'm averaging 50-75 of the buggers at any time. Before that it was 300-500.
|
|
#14
| ||||
| ||||
|
is slurp delay defined in robots.txt? If so, doesn't it affect indexing speed? |
|
#15
| |||
| |||
|
yahoo indexing was notoriously slow for me anyway, I haven't noticed a difference, but maybe that's because I'm submitting a yahoo sitemap too. the problem that I had with Yahoo is that sometimes it'd spike and use up all my mysql connections. My changes above helped my bandwidth resources a little bit too. |
| Thread Tools | |
| |
Similar Threads | ||||
| Thread | Thread Starter | Forum | Replies | Last Post |
| yahoo slurp | xpfUK | General Discussion | 7 | 07-21-2007 11:22 PM |
| Yahoo Slurp bombardment | bjornstrom | General Discussion | 5 | 06-29-2006 03:34 AM |
| increasing yahoo slurp spiders | MentaL | General Discussion | 0 | 01-20-2006 01:08 AM |
| VBSEO and YAHOO Slurp | MentaL | General Discussion | 6 | 11-02-2005 11:11 AM |
| Exclude Yahoo-Slurp by htaccess? | Silmarillion | Analysis: Traffic & SERPS | 5 | 10-19-2005 07:51 PM |