vBulletin 4, the most powerful community software + vBSEO 3.5, the ultimate SEO solution = Your ultimate platform for 2010 and beyond. Click below to learn more.

Page 1 of 2
1 2 LastLast
Results 1 to 15 of 19

Good problem to have with yahoo slurp

This is a discussion on Good problem to have with yahoo slurp within the Analysis: Traffic & SERPS forums, part of the vBulletin SEO Discussion category; Any of you guys have a site that Yahoo Slurp seems to love? I've had problems with max database connections ...

  1. 01-30-2008 09:28 AM #1
    TnTU
    TnTU is offline
    Member
    Real Name
    David
    Join Date
    Jan 2006
    Location
    Nebraska
    Posts
    45

    Good problem to have with yahoo slurp

    Any of you guys have a site that Yahoo Slurp seems to love?

    I've had problems with max database connections whenever Slurp decides to spend more than just a visit at my tntu.net site.

    I'm using a "crawl delay" of 30 in my robots.txt file. I thought that would throttle the beast back a bit, but I don't think it has made much of a dent.

    I currently have 48 users on my forums, 7 members, 10 visitors, 1 google adsense spider, and 30 Slurp agents.

    In a day, I'll get around 900 visits from Slurp.

    Anybody know a good way to calm that spider down?
    Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
    Reply With Quote Reply With Quote  
  2. 01-30-2008 03:29 PM #2
    TnTU
    TnTU is offline
    Member
    Real Name
    David
    Join Date
    Jan 2006
    Location
    Nebraska
    Posts
    45
    I guess a few people have this problem, I'll just keep bumping up that crawl limit for slurp and see if that works...
    yahoo slurp
    Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
    Reply With Quote Reply With Quote  
  3. 03-12-2008 04:36 PM #3
    TnTU
    TnTU is offline
    Member
    Real Name
    David
    Join Date
    Jan 2006
    Location
    Nebraska
    Posts
    45
    i actually got a response from Yahoo the other day, they suggest to increase the crawl delay to 120 or 240 to alleviate the problem, it seems to have worked, hopefully it doesn't affect the Yahoo indexing, as slow as it is already.
    Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
    Reply With Quote Reply With Quote  
  4. 03-12-2008 07:06 PM #4
    Joe Ward
    Joe Ward is offline
    Senior Member
    Real Name
    Joseph Ward
    Join Date
    Jun 2005
    Posts
    23,847
    Blog Entries
    9
    Wow - great that they got back to you about that. Interesting.
    Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
    Reply With Quote Reply With Quote  
  5. 03-12-2008 07:59 PM #5
    Notorious
    Notorious is offline
    Senior Member
    Real Name
    Dhillon
    Join Date
    Apr 2006
    Posts
    327
    Don't be alarmed by the sheer number of Yahoo Slurps,the reason there are lots of them at the same time is because they came from different IPs.
    Like right now I have one Googlebot and 30 Slurps and it’s almost always like that. But at the end of the day, hits from Googlebot far exceed Yahoo Slurp.

    Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
    Reply With Quote Reply With Quote  
  6. 03-14-2008 08:21 PM #6
    amenadiel
    amenadiel is offline
    Senior Member amenadiel's Avatar
    Real Name
    Felipe CHW
    Join Date
    Feb 2007
    Location
    Santiago, Chile
    Posts
    165
    In this very moment, I have the following bots in my "who's online"

    Google (3), Google AdSense (2), Yahoo! Slurp (264), MSNBot (4)




    I've seen more than 400 on some days.
    Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
    Reply With Quote Reply With Quote  
  7. 03-22-2008 02:32 AM #7
    briansol
    briansol is offline
    Senior Member briansol's Avatar
    Real Name
    Brian
    Join Date
    Apr 2006
    Location
    Central CT, USA
    Posts
    6,983
    same here. my current at 2:30 am:

    Yahoo! Slurp (329), Google (2), AskJeeves (1), Google AdSense (1), MSNBot (1)





    I had to modify my maxclients setting because of it.


    [/usr/local/apache/conf/httpd.conf]
    KeepAlive On
    MaxKeepAliveRequests 1000
    KeepAliveTimeout 2
    MinSpareServers 10
    MaxSpareServers 15
    StartServers 10
    MaxClients 200
    MaxRequestsPerChild 1000
    Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
    Reply With Quote Reply With Quote  
  8. 03-22-2008 01:03 PM #8
    KW802
    KW802 is offline
    Senior Member KW802's Avatar
    Real Name
    Kevin
    Join Date
    Jan 2006
    Posts
    163
    There are days that I really hate Yahoo.

    Yahoo! Slurp (412), MSNBot (1), AskJeeves (1), Google (1), Google AdSense (1)
    vBulletin Arcade
    Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
    Reply With Quote Reply With Quote  
  9. 03-23-2008 10:38 AM #9
    amenadiel
    amenadiel is offline
    Senior Member amenadiel's Avatar
    Real Name
    Felipe CHW
    Join Date
    Feb 2007
    Location
    Santiago, Chile
    Posts
    165
    Ok this is getting out of control:

    Google (1), Google AdSense (1), Yahoo! Slurp (521), MSNBot (2)
    Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
    Reply With Quote Reply With Quote  
  10. 03-23-2008 08:14 PM #10
    briansol
    briansol is offline
    Senior Member briansol's Avatar
    Real Name
    Brian
    Join Date
    Apr 2006
    Location
    Central CT, USA
    Posts
    6,983
    Yahoo! Slurp (376), Google AdSense (2), Google (1), AskJeeves (1), MSNBot (1)


    i'm with ya guys... lol
    Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
    Reply With Quote Reply With Quote  
  11. 03-24-2008 11:28 PM #11
    briansol
    briansol is offline
    Senior Member briansol's Avatar
    Real Name
    Brian
    Join Date
    Apr 2006
    Location
    Central CT, USA
    Posts
    6,983
    Yahoo! Slurp (512), Google (2), AskJeeves (1), Google AdSense (2)

    lol
    Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
    Reply With Quote Reply With Quote  
  12. 03-31-2008 05:11 PM #12
    amenadiel
    amenadiel is offline
    Senior Member amenadiel's Avatar
    Real Name
    Felipe CHW
    Join Date
    Feb 2007
    Location
    Santiago, Chile
    Posts
    165
    update:

    Yahoo! Slurp (886), Google (2), MSNBot (3), Google AdSense (1)
    CHW, el mayor sitio en español sobre hardware, overclock y noticias tech

    Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
    Reply With Quote Reply With Quote  
  13. 03-31-2008 05:25 PM #13
    KW802
    KW802 is offline
    Senior Member KW802's Avatar
    Real Name
    Kevin
    Join Date
    Jan 2006
    Posts
    163
    I've since set my Slurp crawl delay to 60 and now I'm averaging 50-75 of the buggers at any time. Before that it was 300-500.
    vBulletin Arcade
    Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
    Reply With Quote Reply With Quote  
  14. 04-01-2008 11:49 AM #14
    amenadiel
    amenadiel is offline
    Senior Member amenadiel's Avatar
    Real Name
    Felipe CHW
    Join Date
    Feb 2007
    Location
    Santiago, Chile
    Posts
    165
    is slurp delay defined in robots.txt?

    If so, doesn't it affect indexing speed?
    CHW, el mayor sitio en español sobre hardware, overclock y noticias tech

    Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
    Reply With Quote Reply With Quote  
  15. 04-01-2008 06:13 PM #15
    TnTU
    TnTU is offline
    Member
    Real Name
    David
    Join Date
    Jan 2006
    Location
    Nebraska
    Posts
    45
    yahoo indexing was notoriously slow for me anyway, I haven't noticed a difference, but maybe that's because I'm submitting a yahoo sitemap too.

    the problem that I had with Yahoo is that sometimes it'd spike and use up all my mysql connections. My changes above helped my bandwidth resources a little bit too.
    Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
    Reply With Quote Reply With Quote  
« another success story | After My URL Rewrite My Indexed Pages Dropped down to 0 »

Similar Threads

  1. yahoo slurp
    By xpfUK in forum General Discussion
    Replies: 7
    Last Post: 07-21-2007, 10:22 PM
  2. Yahoo Slurp bombardment
    By bjornstrom in forum General Discussion
    Replies: 5
    Last Post: 06-29-2006, 02:34 AM
  3. increasing yahoo slurp spiders
    By MentaL in forum General Discussion
    Replies: 0
    Last Post: 01-20-2006, 12:08 AM
  4. VBSEO and YAHOO Slurp
    By MentaL in forum General Discussion
    Replies: 6
    Last Post: 11-02-2005, 10:11 AM
  5. Exclude Yahoo-Slurp by htaccess?
    By Silmarillion in forum Analysis: Traffic & SERPS
    Replies: 5
    Last Post: 10-19-2005, 06:51 PM