Results 1 to 3 of 3

Spider banning

This is a discussion on Spider banning within the General Discussion forums, part of the vBulletin SEO Discussion category; Is there an easy way to stop all spiders except google, msn, yahoo and ask.com via robots text? It seems ...

  1. #1
    Senior Member Lee G's Avatar
    Real Name
    Lee
    Join Date
    Sep 2006
    Location
    Costa Blanca
    Posts
    690
    Liked
    40 times
    Blog Entries
    4

    Spider banning

    Is there an easy way to stop all spiders except google, msn, yahoo and ask.com via robots text?

    It seems that there are so many data gatherers about these days, that offer very little in return, a simple robots text ban would reduce band width, increase page load times and you will still get the same search engine referrals from the likes of Google.
    Possibly better Google ranking if your page loads are a lot faster.

    Any help or feed back on this one would be much appreciated

  2. #2
    Senior Member
    Real Name
    Ceri May
    Join Date
    Jul 2009
    Location
    United Kingdom
    Posts
    1,726
    Liked
    15 times
    Blog Entries
    1
    The sort of gathers you are talking about tend to ignore robots.txt files so it would be no good... Don't get me wrong it is possible to deny * then set google, msn, yahoo and ask to allow but I am not confident it will help in your situation.

    The only way to block rouge bots is to ban then via IP address or Useragent in your .htaccess file HOWEVER this can easily be messed up and end up blocking legitimate bots or users and the more you add to you .htaccess file the longer it will take to process and would actually end up effecting site speed if you let it get too big.

  3. #3
    vBSEO Staff Brian Cummiskey's Avatar
    Real Name
    Brian Cummiskey
    Join Date
    Jul 2009
    Location
    btwn NYC and Boston
    Posts
    12,789
    Liked
    657 times
    Blog Entries
    2
    It's basically a waste of time.

    You basically have 2 options--
    set a whitelist, block everything else
    set a blacklist, block everything else

    Either way, you need to account for ALL good things, or ALL bad things.
    Something that is simply not possible.

Similar Threads

  1. Banning Baidu through htaccess?
    By wac in forum General Discussion
    Replies: 13
    Last Post: 04-22-2010, 05:56 PM
  2. Google Spider / Google Adsense Spider
    By Yours Truly in forum General Discussion
    Replies: 2
    Last Post: 02-29-2008, 04:34 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •