vBulletin 4, the most powerful community software + vBSEO 3.5, the ultimate SEO solution = Your ultimate platform for 2010 and beyond. Click below to learn more.

Results 1 to 9 of 9

Yahoo Slurp Spider emailing members?

This is a discussion on Yahoo Slurp Spider emailing members? within the Troubleshooting forums, part of the vBSEO Google/Yahoo Sitemap category; I noticed several of the Yahoo spiders emailing members... of course it was getting a red stop sign... but it's ...

  1. #1
    Senior Member Sonnie's Avatar
    Real Name
    Sonnie
    Join Date
    May 2006
    Location
    L.A. (Lower Alabama)
    Posts
    208

    Yahoo Slurp Spider emailing members?

    I noticed several of the Yahoo spiders emailing members... of course it was getting a red stop sign... but it's wasting time going through all my members trying to email them.

    How can I prevent this?

    Thanks!

  2. #2
    vBSEO Staff Oleg Ignatiuk's Avatar
    Real Name
    Oleg Ignatiuk
    Join Date
    Jun 2005
    Location
    Belarus
    Posts
    24,543

    Re: Yahoo Slurp Spider emailing members?

    Hello Sonnie,

    perhaps adding the following to robots.txt will prevent yahoo bot from doing so:
    Code:
    Disallow: /forums/sendmessage.php
    
    Btw, I just noticed that all your entries in robots.txt are looking like "Disallow: /ajax.php" etc, while your vBulletin folder is /forums/. You should include it into every line like in my example above.
    Oleg Ignatiuk / Crawlability Inc.
    Sneek Preview Video of the new Control Panel

    vBSEO 3.5 RC2 (Final)- Released for your Evaluation


  3. #3
    Senior Member Sonnie's Avatar
    Real Name
    Sonnie
    Join Date
    May 2006
    Location
    L.A. (Lower Alabama)
    Posts
    208

    Re: Yahoo Slurp Spider emailing members?

    Thanks Oleg,

    Yeah... I had /sendmessage.php in there but didn't have the /forums/ in there.

    My robots.txt file is in my root directory and there are some root folders and files that I assume I won't need to insert the /forums/ in front of. I also assume I need to keep the robots.txt file in the root and not move it to /forums. Am I assuming all this correctly?

  4. #4
    vBSEO Staff Oleg Ignatiuk's Avatar
    Real Name
    Oleg Ignatiuk
    Join Date
    Jun 2005
    Location
    Belarus
    Posts
    24,543

    Re: Yahoo Slurp Spider emailing members?

    Yes, confirming - you should keep robots.txt in the domain root and add /forums/ for the vB pages definition.
    Oleg Ignatiuk / Crawlability Inc.
    Sneek Preview Video of the new Control Panel

    vBSEO 3.5 RC2 (Final)- Released for your Evaluation


  5. #5
    Senior Member Sonnie's Avatar
    Real Name
    Sonnie
    Join Date
    May 2006
    Location
    L.A. (Lower Alabama)
    Posts
    208

    Re: Yahoo Slurp Spider emailing members?

    Thank you kindly... I think I got it all fixed now... hopefully.

  6. #6
    vBSEO Staff Oleg Ignatiuk's Avatar
    Real Name
    Oleg Ignatiuk
    Join Date
    Jun 2005
    Location
    Belarus
    Posts
    24,543

    Re: Yahoo Slurp Spider emailing members?

    Great!
    Oleg Ignatiuk / Crawlability Inc.
    Sneek Preview Video of the new Control Panel

    vBSEO 3.5 RC2 (Final)- Released for your Evaluation


  7. #7
    Junior Member
    Real Name
    Sherm Homan
    Join Date
    Jul 2008
    Posts
    2
    I hate to revive a zombie thread, but I searched, honest I did! This is the only good piece of advice that I could find on the subject of Yahoo slurps attempting to email vBulletin members. I had never heard of it before today, but I saw it, it does happen. My question to any of the posters of this thread who are still alive (!)
    Did this work?
    Code:
    Disallow: /forums/sendmessage.php
    
    Thanks!

  8. #8
    Senior Member briansol's Avatar
    Real Name
    Brian
    Join Date
    Apr 2006
    Location
    Central CT, USA
    Posts
    6,983
    robots is merely a suggestion.

    you should enable captcha or another human verification for guests on the email system.

  9. #9
    Junior Member
    Real Name
    Sherm Homan
    Join Date
    Jul 2008
    Posts
    2
    Thanks, Brian! We are using captcha and email send/receive verification. The slurps are not registering as users, they are attempting and failing to send email to our users.

Similar Threads

  1. yahoo slurp
    By xpfUK in forum General Discussion
    Replies: 7
    Last Post: 07-21-2007, 10:22 PM
  2. increasing yahoo slurp spiders
    By MentaL in forum General Discussion
    Replies: 0
    Last Post: 01-20-2006, 12:08 AM
  3. VBSEO and YAHOO Slurp
    By MentaL in forum General Discussion
    Replies: 6
    Last Post: 11-02-2005, 10:11 AM