Results 1 to 14 of 14

Banning Baidu through htaccess?

This is a discussion on Banning Baidu through htaccess? within the General Discussion forums, part of the vBSEO SEO Plugin category; How to ban Baidu through .htaccess and where exactly do I put this code in .htaccess now that I have ...

  1. #1
    wac
    wac is offline
    Senior Member Array
    Real Name
    toto
    Join Date
    Apr 2009
    Posts
    134
    Liked
    0 times

    Banning Baidu through htaccess?

    How to ban Baidu through .htaccess and where exactly do I put this code in .htaccess now that I have vbseo? I only get about 10 visitors a month from China anyway and I have close to 15 bots from Baidu all over the site all day for months now. These are just a few of their bots. If I have to ban China entirely is fine with me.

    baiduspider-123-125-66-76.crawl.baidu.com
    baiduspider-123-125-66-93.crawl.baidu.com
    baiduspider-123-125-66-66.crawl.baidu.com

  2. #2
    vBSEO Staff Array Brian Cummiskey's Avatar
    Real Name
    Brian Cummiskey
    Join Date
    Jul 2009
    Location
    btwn NYC and Boston
    Posts
    12,789
    Liked
    675 times
    Blog Entries
    2
    Code:
    RewriteCond %{HTTP_USER_AGENT} ^baiduspider [NC] 
    RewriteRule .* - [F]
    should do the trick
    Brian Cummiskey / Crawlability Inc.
    Security bulletin - Patch Level for all supported versions released

    Unveiling the NEW vBSEO Sitemap Generator 3.0. - available NOW for vBSEO Customers!


  3. #3
    wac
    wac is offline
    Senior Member Array
    Real Name
    toto
    Join Date
    Apr 2009
    Posts
    134
    Liked
    0 times
    Quote Originally Posted by Brian Cummiskey View Post
    Code:
    RewriteCond %{HTTP_USER_AGENT} ^baiduspider [NC] 
    RewriteRule .* - [F]
    should do the trick
    Thanks for that Brian. I suppose I have to put that following the Vbseo.com code right at the bottom right? or at the top? Sorry I have no clue about this and have screwed up this file before and have had site problems.

  4. #4
    vBSEO Staff Array Brian Cummiskey's Avatar
    Real Name
    Brian Cummiskey
    Join Date
    Jul 2009
    Location
    btwn NYC and Boston
    Posts
    12,789
    Liked
    675 times
    Blog Entries
    2
    That should go in your htacess file, right above the sitemap rule
    Brian Cummiskey / Crawlability Inc.
    Security bulletin - Patch Level for all supported versions released

    Unveiling the NEW vBSEO Sitemap Generator 3.0. - available NOW for vBSEO Customers!


  5. #5
    wac
    wac is offline
    Senior Member Array
    Real Name
    toto
    Join Date
    Apr 2009
    Posts
    134
    Liked
    0 times
    Great. Thanks a lot Brian.

  6. #6
    wac
    wac is offline
    Senior Member Array
    Real Name
    toto
    Join Date
    Apr 2009
    Posts
    134
    Liked
    0 times
    Brian, the Baidu bot (baiduspider-220-181-7-64.crawl.baidu.com) is getting in somehow. Can you please look at my .htaccess file below? This is my exact .htaccess file. Thanks for your time.

    # Comment the following line (add '#' at the beginning)
    # to disable mod_rewrite functions.
    # Please note: you still need to disable the hack in
    # the vBSEO control panel to stop url rewrites.
    RewriteEngine On

    # Some servers require the Rewritebase directive to be
    # enabled (remove '#' at the beginning to activate)
    # Please note: when enabled, you must include the path
    # to your root vB folder (i.e. RewriteBase /forums/)
    #RewriteBase /

    #RewriteCond %{HTTP_HOST} !^www\.yourdomain\.com
    #RewriteRule (.*) http://www.yourdomain.com/forums/$1 [L,R=301]

    RewriteCond %{HTTP_USER_AGENT} ^baiduspider [NC]
    RewriteRule .* - [F]

    RewriteRule ^((urllist|sitemap_).*\.(xml|txt)(\.gz)?)$ vbseo_sitemap/vbseo_getsitemap.php?sitemap=$1 [L]

    RewriteCond %{REQUEST_URI} !(admincp/|modcp/|cron|vbseo_sitemap)
    RewriteRule ^((archive/)?(.*\.php(/.*)?))$ vbseo.php [L,QSA]

    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteCond %{REQUEST_FILENAME} !-d
    RewriteCond %{REQUEST_FILENAME} !/(admincp|modcp|clientscript|cpstyles|images)/
    RewriteRule ^(.+)$ vbseo.php [L,QSA]

  7. #7
    vBSEO Staff Array Brian Cummiskey's Avatar
    Real Name
    Brian Cummiskey
    Join Date
    Jul 2009
    Location
    btwn NYC and Boston
    Posts
    12,789
    Liked
    675 times
    Blog Entries
    2
    the code is correct. Perhaps it's spoofing it's useragent?

    I'm not sure if it will still record a hit. with going to [F] which basically means fail, it will still be on your site, but will see a 401
    Brian Cummiskey / Crawlability Inc.
    Security bulletin - Patch Level for all supported versions released

    Unveiling the NEW vBSEO Sitemap Generator 3.0. - available NOW for vBSEO Customers!


  8. #8
    wac
    wac is offline
    Senior Member Array
    Real Name
    toto
    Join Date
    Apr 2009
    Posts
    134
    Liked
    0 times
    Brian what would the code look like now if I want to ban an ip address from another bot along with the baidu spider? The ip address from apparently a forum scraper s below:

    142.166.170.100

    and this is what I have now:

    RewriteCond %{HTTP_USER_AGENT} ^baiduspider [NC]
    RewriteRule .* - [F]

  9. #9
    vBSEO Staff Array Brian Cummiskey's Avatar
    Real Name
    Brian Cummiskey
    Join Date
    Jul 2009
    Location
    btwn NYC and Boston
    Posts
    12,789
    Liked
    675 times
    Blog Entries
    2
    You can add it:

    order allow,deny
    deny from 142.166.170.100
    allow from all


    right under:
    RewriteRule .* - [F]
    Brian Cummiskey / Crawlability Inc.
    Security bulletin - Patch Level for all supported versions released

    Unveiling the NEW vBSEO Sitemap Generator 3.0. - available NOW for vBSEO Customers!


  10. #10
    Member Array
    Real Name
    Steven
    Join Date
    Jan 2009
    Location
    UK
    Posts
    55
    Liked
    0 times
    Hello, Sorry to but in.

    I've had this bot all over my site for ages, should I be banning it?

  11. #11
    Senior Member Array Lee G's Avatar
    Real Name
    Lee
    Join Date
    Sep 2006
    Location
    Costa Blanca
    Posts
    695
    Liked
    43 times
    Blog Entries
    4
    Baidu is one of Chinas main search engines now that Google have pulled out.
    So if your after hits, it can still be of use.
    I was monitoring the hits from China on my own forum and was not impressed with what most hits from China, were trying to do.
    So I banned the whole continent.
    I tried robots text etc to reduce Baidu hitting my forum and got fed up in the end.

  12. #12
    Senior Member Array
    Real Name
    Christian Thiessen
    Join Date
    May 2007
    Posts
    101
    Liked
    0 times
    Blog Entries
    1
    Hi,
    best thing to stop Chinese Traffic is to write an article of the Human Rights in China and Tibet.
    The Chinese Firewall will do the Rest.

    Christian

  13. #13
    Member Array
    Real Name
    Jaffery
    Join Date
    Dec 2009
    Posts
    97
    Liked
    0 times
    Quote Originally Posted by CThiessen View Post
    Hi,
    best thing to stop Chinese Traffic is to write an article of the Human Rights in China and Tibet.
    The Chinese Firewall will do the Rest.

    Christian
    LOLz

    Can't resist to say it !

  14. #14
    wac
    wac is offline
    Senior Member Array
    Real Name
    toto
    Join Date
    Apr 2009
    Posts
    134
    Liked
    0 times
    Quote Originally Posted by CThiessen View Post
    Hi,
    best thing to stop Chinese Traffic is to write an article of the Human Rights in China and Tibet.
    The Chinese Firewall will do the Rest.

    Christian
    Reminds me of that story of the semitruck stuck underneath a bridge and they had a lot of construction workers, mechanics, policemen, firefighters trying to get the truck unstuck, trying to pull it out unsuccessfully for hours, and there came a little boy and said to the policeman, why don't you guys take the air out of the tires? LOL! Policeman went running to the guys laughing and they were able to pull the truck out. I think it was a true story.

    LOL. I'll create a post on human rights to see if it works!

Similar Threads

  1. Replies: 2
    Last Post: 12-24-2009, 04:52 AM
  2. Replies: 5
    Last Post: 05-12-2009, 03:29 PM
  3. .htaccess in forum root w/ exisiting .htaccess?
    By OttoBomb in forum Troubleshooting
    Replies: 5
    Last Post: 07-03-2007, 03:45 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •