Results 1 to 4 of 4

Yahoo Bot Crawl Delay: Any harm?

This is a discussion on Yahoo Bot Crawl Delay: Any harm? within the General Discussion forums, part of the vBulletin SEO Discussion category; Hello, we still have ~400 Yahoo Bots online at our site, which generally spoken is a good thing. Even with ...

  1. #1
    Senior Member Lian's Avatar
    Real Name
    Lian
    Join Date
    Jun 2006
    Location
    Munich, DE
    Posts
    333
    Liked
    0 times

    Question Yahoo Bot Crawl Delay: Any harm?

    Hello,

    we still have ~400 Yahoo Bots online at our site, which generally spoken is a good thing.
    Even with their new Bot recently relaunched we have that many bots online.

    But the performance impact is noticable, so we think of placing a crawl delay for the bot in our robots.txt, see:
    Yahoo! Help -
    We think of
    Code:
    User-agent: Slurp
    Crawl-delay: 5
    Will that do any harm SEO-wise?

    Will it help reducing the queries of Y Bot to reduce our server's load?
    I guess so...

    Thanks,
    Lian

  2. #2
    Senior Member Lian's Avatar
    Real Name
    Lian
    Join Date
    Jun 2006
    Location
    Munich, DE
    Posts
    333
    Liked
    0 times
    Found it:
    Setting a crawl-delay may limit the coverage and freshness of your content representation in Yahoo! search results. If you do feel that a crawl-delay is necessary, use small values to avoid blocking Slurp discovery and refresh of your key content.
    Nope, we surely don't want that.

  3. #3
    Senior Member
    Real Name
    Joseph Ward
    Join Date
    Jun 2005
    Posts
    23,845
    Liked
    36 times
    Blog Entries
    9
    FYI Yahoo released a new version of Slurp. It's supported to be more efficient in crawling and aims to reduce bandwidth consumption (and performance impact):

    http://www.vbseo.com/f104/yahoo-slur...released-6588/

  4. #4
    Senior Member Lian's Avatar
    Real Name
    Lian
    Join Date
    Jun 2006
    Location
    Munich, DE
    Posts
    333
    Liked
    0 times
    Thanks, I've read the thread.

    I wonder why we still have ~400 online during (bot) peak times...

Similar Threads

  1. Yahoo Slurp - New Version of Search Engine Crawler Released
    By Joe Ward in forum General Discussion
    Replies: 4
    Last Post: 08-16-2006, 03:35 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •