Results 1 to 8 of 8

yahoo slurp

This is a discussion on yahoo slurp within the General Discussion forums, part of the vBulletin SEO Discussion category; Hi all Can anyone tell me how to restrict the Yahoo! Slurp spider. Google and MSN Bots are fine, they ...

  1. #1
    Member
    Real Name
    mal
    Join Date
    Mar 2006
    Posts
    81
    Liked
    0 times

    Question yahoo slurp

    Hi all

    Can anyone tell me how to restrict the Yahoo! Slurp spider.

    Google and MSN Bots are fine, they come, crawl and leave.
    Yahoo! Slurp however is a constant presence and will not go away.
    Can be up to 20 listed in Who's online 24/7 .

    vBulletin 3.5.4 - vBSEO 2.4.0

  2. #2
    Senior Member
    Real Name
    Keith Cohen
    Join Date
    Jul 2005
    Location
    Raleigh, NC USA
    Posts
    6,147
    Liked
    12 times
    Hello Mal:

    You can add this to your robots.txt to block Yahoo:

    User-agent: Slurp
    Disallow: /

  3. #3
    Member
    Real Name
    mal
    Join Date
    Mar 2006
    Posts
    81
    Liked
    0 times
    Hi Keith

    I'd rather not block it completely, just restrict it .

  4. #4
    Senior Member TECK's Avatar
    Real Name
    Floren Munteanu
    Join Date
    Apr 2006
    Location
    Canada
    Posts
    788
    Liked
    0 times
    Blog Entries
    2
    Did you tried rel="nofollow" on your link? It should stop any crawler to follow that specific link.

  5. #5
    Member
    Real Name
    mal
    Join Date
    Mar 2006
    Posts
    81
    Liked
    0 times
    The problem is that that they just sit there on the index page .

    The robots.text entry has blocked them for now, however I don't see this as the long term answer.
    The others come and go, what is it with Slurp :(
    It's like they have decided to squat . lol

  6. #6
    Senior Member
    Real Name
    Keith Cohen
    Join Date
    Jul 2005
    Location
    Raleigh, NC USA
    Posts
    6,147
    Liked
    12 times
    I would suggest just letting them do their thing. If they are just sitting there, they aren't using your bandwidth, so you shouldn't have anything to worry about.

  7. #7
    Member
    Real Name
    mal
    Join Date
    Mar 2006
    Posts
    81
    Liked
    0 times
    Just an update on this.

    slurp is still camped out on my site. Have had some advice saying it may well be related to the RSS/XML feed.

    Anyway, having received a few "too many connections" errors, I added this to my robots.txt file


    Code:
     
    User-agent: Slurp 
    Crawl-delay: 5
    No errors since.

  8. #8
    Junior Member
    Join Date
    Nov 2005
    Posts
    26
    Liked
    0 times
    I've upped my crawl delay to 60, Yahoo need to sort out their crawler, it's going nuts on my site.

Similar Threads

  1. Yahoo doesnt like my site...
    By Barry in forum General Discussion
    Replies: 6
    Last Post: 02-04-2006, 03:24 AM
  2. increasing yahoo slurp spiders
    By MentaL in forum General Discussion
    Replies: 0
    Last Post: 01-20-2006, 01:08 AM
  3. Why does Yahoo hate me?
    By caliman in forum Off-Topic & Chit Chat
    Replies: 5
    Last Post: 11-18-2005, 03:31 AM
  4. Yahoo not reconigzing the vbseo urls
    By emin in forum General Discussion
    Replies: 5
    Last Post: 11-14-2005, 09:05 PM
  5. VBSEO and YAHOO Slurp
    By MentaL in forum General Discussion
    Replies: 6
    Last Post: 11-02-2005, 11:11 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •