Results 1 to 9 of 9

Yahoo Slurp Spider emailing members?

This is a discussion on Yahoo Slurp Spider emailing members? within the Troubleshooting forums, part of the vBSEO Google/Yahoo Sitemap category; I noticed several of the Yahoo spiders emailing members... of course it was getting a red stop sign... but it's ...

  1. #1
    Senior Member Sonnie's Avatar
    Real Name
    Sonnie
    Join Date
    May 2006
    Location
    L.A. (Lower Alabama)
    Posts
    305
    Liked
    2 times

    Yahoo Slurp Spider emailing members?

    I noticed several of the Yahoo spiders emailing members... of course it was getting a red stop sign... but it's wasting time going through all my members trying to email them.

    How can I prevent this?

    Thanks!

  2. #2
    vBSEO Staff Oleg Ignatiuk's Avatar
    Real Name
    Oleg Ignatiuk
    Join Date
    Jun 2005
    Location
    Belarus
    Posts
    25,689
    Liked
    157 times
    Hello Sonnie,

    perhaps adding the following to robots.txt will prevent yahoo bot from doing so:
    Code:
    Disallow: /forums/sendmessage.php
    Btw, I just noticed that all your entries in robots.txt are looking like "Disallow: /ajax.php" etc, while your vBulletin folder is /forums/. You should include it into every line like in my example above.
    Oleg Ignatiuk / Crawlability Inc.
    vBSEO 3.6.0 GOLD Released!
    Unveiling the NEW vBSEO Sitemap Generator 3.0. - available NOW for vBSEO Customers!


  3. #3
    Senior Member Sonnie's Avatar
    Real Name
    Sonnie
    Join Date
    May 2006
    Location
    L.A. (Lower Alabama)
    Posts
    305
    Liked
    2 times
    Thanks Oleg,

    Yeah... I had /sendmessage.php in there but didn't have the /forums/ in there.

    My robots.txt file is in my root directory and there are some root folders and files that I assume I won't need to insert the /forums/ in front of. I also assume I need to keep the robots.txt file in the root and not move it to /forums. Am I assuming all this correctly?

  4. #4
    vBSEO Staff Oleg Ignatiuk's Avatar
    Real Name
    Oleg Ignatiuk
    Join Date
    Jun 2005
    Location
    Belarus
    Posts
    25,689
    Liked
    157 times
    Yes, confirming - you should keep robots.txt in the domain root and add /forums/ for the vB pages definition.
    Oleg Ignatiuk / Crawlability Inc.
    vBSEO 3.6.0 GOLD Released!
    Unveiling the NEW vBSEO Sitemap Generator 3.0. - available NOW for vBSEO Customers!


  5. #5
    Senior Member Sonnie's Avatar
    Real Name
    Sonnie
    Join Date
    May 2006
    Location
    L.A. (Lower Alabama)
    Posts
    305
    Liked
    2 times
    Thank you kindly... I think I got it all fixed now... hopefully.

  6. #6
    vBSEO Staff Oleg Ignatiuk's Avatar
    Real Name
    Oleg Ignatiuk
    Join Date
    Jun 2005
    Location
    Belarus
    Posts
    25,689
    Liked
    157 times
    Great!
    Oleg Ignatiuk / Crawlability Inc.
    vBSEO 3.6.0 GOLD Released!
    Unveiling the NEW vBSEO Sitemap Generator 3.0. - available NOW for vBSEO Customers!


  7. #7
    Junior Member
    Real Name
    Sherm Homan
    Join Date
    Jul 2008
    Posts
    2
    Liked
    0 times
    I hate to revive a zombie thread, but I searched, honest I did! This is the only good piece of advice that I could find on the subject of Yahoo slurps attempting to email vBulletin members. I had never heard of it before today, but I saw it, it does happen. My question to any of the posters of this thread who are still alive (!)
    Did this work?
    Code:
    Disallow: /forums/sendmessage.php
    Thanks!

  8. #8
    Senior Member briansol's Avatar
    Real Name
    Brian
    Join Date
    Apr 2006
    Location
    Central CT, USA
    Posts
    6,981
    Liked
    8 times
    robots is merely a suggestion.

    you should enable captcha or another human verification for guests on the email system.

  9. #9
    Junior Member
    Real Name
    Sherm Homan
    Join Date
    Jul 2008
    Posts
    2
    Liked
    0 times
    Thanks, Brian! We are using captcha and email send/receive verification. The slurps are not registering as users, they are attempting and failing to send email to our users.

Similar Threads

  1. yahoo slurp
    By xpfUK in forum General Discussion
    Replies: 7
    Last Post: 07-21-2007, 11:22 PM
  2. increasing yahoo slurp spiders
    By MentaL in forum General Discussion
    Replies: 0
    Last Post: 01-20-2006, 01:08 AM
  3. VBSEO and YAHOO Slurp
    By MentaL in forum General Discussion
    Replies: 6
    Last Post: 11-02-2005, 11:11 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •