Baiduspider, how to tame spiders and bots

I will admit I am lousy at trying to figure this stuff out. But I have at least 100 of the baiduspiders on my site at all given times.

How do I block them, or at least TAME them, via robots.txt without blocking the good spiders?

Any help would be appreciated. Thanks

What do you want to do? block or slow down?

I think for the time being I want to block them, at least for a little while, so I can see how many real guests I have at my site. Anyway you could tell me how to do both, so I can change it in the future if I decide to?

Thank you

add

User-agent: Baiduspider
Disallow: /

to your robots.txt file


and

Code:

deny from .baidu.com

to your htaccess file directly after rewriteengine on

to tame, you would use the cralw-delay setting in robots.

Thanks Brian, where do I find my .htaccess file again? The only one I found is in my forum root, (the one that was provided by vbseo), but it still says 'mysite.com', was I supposed to change that to englishbulldognews.com?

Yes, it should be in forum root.

# denotes a comment.

see
[How to] Setup www.yourdomain.com or yourdomain.com (www vs. non-www) - With .htaccess 301 redirects - vBulletin SEO Forums

Here is my robots and .htaccess, are they correct?

User-agent: Baiduspider
Disallow: /

User-agent: *
Allow: /

User-agent: *
Disallow: /forums/clientscript/
Disallow: /forums/cpstyles/
Disallow: /forums/customavatars/
Disallow: /forums/customprofilepics/
Disallow: /forums/ajax.php
Disallow: /forums/arcade.php
Disallow: /forums/attachment.php
Disallow: /forums/awards.php
Disallow: /forums/calendar.php
Disallow: /forums/cron.php
Disallow: /forums/editpost.php
Disallow: /forums/faq.php
Disallow: /forums/global.php
Disallow: /forums/image.php
Disallow: /forums/inlinemod.php
Disallow: /forums/joinrequests.php
Disallow: /forums/login.php
Disallow: /forums/market.php
Disallow: /forums/market_bank.php
Disallow: /forums/market_convert.php
Disallow: /forums/member.php
Disallow: /forums/memberlist.php
Disallow: /forums/misc.php
Disallow: /forums/moderator.php
Disallow: /forums/newattachment.php
Disallow: /forums/newreply.php
Disallow: /forums/newthread.php
Disallow: /forums/online.php
Disallow: /forums/poll.php
Disallow: /forums/postings.php
Disallow: /forums/printthread.php
Disallow: /forums/private.php
Disallow: /forums/profile.php
Disallow: /forums/register.php
Disallow: /forums/report.php
Disallow: /forums/reputation.php
Disallow: /forums/sendmessage.php
Disallow: /forums/showgroups.php
Disallow: /forums/subscription.php
Disallow: /forums/threadrate.php
Disallow: /forums/usercp.php
Disallow: /forums/usernote.php
Disallow: /forums/vbactivity.php
Disallow: *.js
Disallow: /forums/includes/
Disallow: /forums/install/
Disallow: /forums/customavatars/
Disallow: /css/



sitemap: http://www.englishbulldognews.com/fo...p_index.xml.gz

# Comment the following line (add '#' at the beginning)
# to disable mod_rewrite functions.
# Please note: you still need to disable the hack in
# the vBSEO control panel to stop url rewrites.
RewriteEngine On

# Some servers require the Rewritebase directive to be
# enabled (remove '#' at the beginning to activate)
# Please note: when enabled, you must include the path
# to your root vB folder (i.e. RewriteBase /forums/)
#RewriteBase /

RewriteCond %{HTTP_HOST} !^englishbulldognews\.com$
RewriteRule ^(.*)$ http://englishbulldognews.com/$1 [L,R=301]

RewriteRule ^((urllist|sitemap_).*\.(xml|txt)(\.gz)?)$ vbseo_sitemap/vbseo_getsitemap.php?sitemap=$1 [L]

RewriteCond %{REQUEST_URI} !(admincp/|modcp/|cron|vbseo_sitemap)
RewriteRule ^((archive/)?(.*\.php(/.*)?))$ vbseo.php [L,QSA]

RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME} !/(admincp|modcp|clientscript|cpstyles|images)/
RewriteRule ^(.+)$ vbseo.php [L,QSA]

RewriteEngine On
RewriteRule ^((urllist|sitemap).*\.(xml|txt)(\.gz)?)$ vbseo_sitemap/vbseo_getsitemap.php?sitemap=$1 [L]

RewriteEngine On
deny from .baidu.com

delete
User-agent: *
Allow: /

as it is unnecessary. that is default behavoir.

Okay thank you So far the bots are still there tho. Does it take a day or so to go through?

They will still hit your server. They just see this fail message now. hopefully, the bot gets the hint eventually.

If you don't want it to hit your server, you need to block it at the firewall level, which you likely don't have access to configure.

Yes, but I was seeing them on my forums, which are not attached to my home page. However, they seem to be off this morning. Now I just got a new spider instead, dont know who/what it is. 124.115.0.*