I noticed several of the Yahoo spiders emailing members... of course it was getting a red stop sign... but it's wasting time going through all my members trying to email them.
How can I prevent this?
Thanks!
This is a discussion on Yahoo Slurp Spider emailing members? within the Troubleshooting forums, part of the vBSEO Google/Yahoo Sitemap category; I noticed several of the Yahoo spiders emailing members... of course it was getting a red stop sign... but it's ...
I noticed several of the Yahoo spiders emailing members... of course it was getting a red stop sign... but it's wasting time going through all my members trying to email them.
How can I prevent this?
Thanks!
Hello Sonnie,
perhaps adding the following to robots.txt will prevent yahoo bot from doing so:
Btw, I just noticed that all your entries in robots.txt are looking like "Disallow: /ajax.php" etc, while your vBulletin folder is /forums/. You should include it into every line like in my example above.Code:Disallow: /forums/sendmessage.php
Oleg Ignatiuk / Crawlability Inc.
vBSEO 3.6.0 GOLD Released!
Unveiling the NEW vBSEO Sitemap Generator 3.0. - available NOW for vBSEO Customers!
Thanks Oleg,
Yeah... I had /sendmessage.php in there but didn't have the /forums/ in there.
My robots.txt file is in my root directory and there are some root folders and files that I assume I won't need to insert the /forums/ in front of. I also assume I need to keep the robots.txt file in the root and not move it to /forums. Am I assuming all this correctly?
Yes, confirming - you should keep robots.txt in the domain root and add /forums/ for the vB pages definition.
Oleg Ignatiuk / Crawlability Inc.
vBSEO 3.6.0 GOLD Released!
Unveiling the NEW vBSEO Sitemap Generator 3.0. - available NOW for vBSEO Customers!
Thank you kindly... I think I got it all fixed now... hopefully.
Great!![]()
Oleg Ignatiuk / Crawlability Inc.
vBSEO 3.6.0 GOLD Released!
Unveiling the NEW vBSEO Sitemap Generator 3.0. - available NOW for vBSEO Customers!
I hate to revive a zombie thread, but I searched, honest I did! This is the only good piece of advice that I could find on the subject of Yahoo slurps attempting to email vBulletin members. I had never heard of it before today, but I saw it, it does happen. My question to any of the posters of this thread who are still alive (!)
Did this work?Thanks!Code:Disallow: /forums/sendmessage.php
robots is merely a suggestion.
you should enable captcha or another human verification for guests on the email system.
Thanks, Brian! We are using captcha and email send/receive verification. The slurps are not registering as users, they are attempting and failing to send email to our users.