Results 1 to 10 of 10

Which SEO BOTS to block to save bandwidth?

This is a discussion on Which SEO BOTS to block to save bandwidth? within the General Discussion forums, part of the vBulletin SEO Discussion category; Hi ive been told there are dirty bots that use up your bandwidth and its best to block some. Can ...

  1. #1
    Member Array
    Real Name
    maksam
    Join Date
    Apr 2008
    Posts
    80
    Liked
    0 times

    Which SEO BOTS to block to save bandwidth?

    Hi ive been told there are dirty bots that use up your bandwidth and its best to block some.

    Can the experts of vbSEO tell me or advise me rather on the most popular bots "not to block" i.e Google, Yahoo, Msn etc... what are the other top SEO bots not to block?

    Could you also give me the correct string for the bots that i should enable server wide? As we know there are two different types of google and yahoo bots. Yahoo, Yahoo Slurp etc.. and also we have Google, Google Adsense bots etc.

    In otherwords should we be even blocking these bots or the more the merrier?

    Any advise would be appreciated.

  2. #2
    Senior Member Array
    Real Name
    Joseph Ward
    Join Date
    Jun 2005
    Posts
    23,845
    Liked
    47 times
    Blog Entries
    9
    Here is an interesting resource with a list of suggestions for bots to block:
    Block bad bots from your site

  3. #3
    Member Array
    Real Name
    maksam
    Join Date
    Apr 2008
    Posts
    80
    Liked
    0 times
    Quote Originally Posted by Joe Ward View Post
    Here is an interesting resource with a list of suggestions for bots to block:
    Block bad bots from your site

    Im assuming there are more bots to block as oppose to allow, would you be able to tell me the number other bots to have other than the usuall ones?

    How can we distinguish from what is a bad/good bot and whether to allow it or not...?

    Cheers

  4. #4
    Senior Member Array
    Real Name
    Joseph Ward
    Join Date
    Jun 2005
    Posts
    23,845
    Liked
    47 times
    Blog Entries
    9
    The obvious ones are Google, Yahoo, & MSN. However, you might want to simply check your referrers. Find all search engines that have sent you any meaningful traffic, and then consider adding them to your approve list.

    Here is an interesting resource you should check out:
    What Do SEO/SEM People Put In Robots.txt Files? | Hobo

  5. #5
    Member Array
    Real Name
    maksam
    Join Date
    Apr 2008
    Posts
    80
    Liked
    0 times
    My last query...

    I understand this is via Robots.txt, my robots.txt is open to all.. however what can be done if my host is blocking robots via server? Their excuse is that to save me bandwidth and to eliminate rogue bots which are "dirty"?

    Can they even do this via server?

  6. #6
    Senior Member Array
    Real Name
    Joseph Ward
    Join Date
    Jun 2005
    Posts
    23,845
    Liked
    47 times
    Blog Entries
    9
    Well - there are two things your host could do in this regard:

    a) They could actually place a robots.txt in client accounts, or revise yours.
    b) They can simply block traffic from the bots via IP at the server software level.

    While this may actually be beneficial to you in some cases, it could be a serious problem in other cases, such as if the host decides to block Googlebot from all clients hosted! It has happened:

    I would ask them what method they are using the block the bots, and I would ask for a full list of everything blocked.

    If the list is unacceptable to you, such as the blocking of a major crawler like Googlebot, I would change hosting providers. The reason I say "change providers" and NOT "ask them to stop doing it" is because:

    They already did you a disservice (in the case of Googlebot blocking), and since there are so many other providers out there, you do not need to feel compelled to give them a 2nd chance.
    Highly Traffic Drop

    Some hosting info:
    vBSEO Web Hosting Survey - Demand Blazing Speed!

  7. #7
    Member Array
    Real Name
    maksam
    Join Date
    Apr 2008
    Posts
    80
    Liked
    0 times
    Quote Originally Posted by Joe Ward View Post
    Well - there are two things your host could do in this regard:

    a) They could actually place a robots.txt in client accounts, or revise yours.
    b) They can simply block traffic from the bots via IP at the server software level.

    While this may actually be beneficial to you in some cases, it could be a serious problem in other cases, such as if the host decides to block Googlebot from all clients hosted! It has happened:

    I would ask them what method they are using the block the bots, and I would ask for a full list of everything blocked.

    If the list is unacceptable to you, such as the blocking of a major crawler like Googlebot, I would change hosting providers. The reason I say "change providers" and NOT "ask them to stop doing it" is because:

    They already did you a disservice (in the case of Googlebot blocking), and since there are so many other providers out there, you do not need to feel compelled to give them a 2nd chance.
    Highly Traffic Drop

    Some hosting info:
    vBSEO Web Hosting Survey - Demand Blazing Speed!

    Cheers for that, yes they never told me about this. Because i noticed a huge drop in bots and had a few htaccess issues i thought it was htaccess. So i queried with them and i explained htaccess could be a problem as i may of done the url rewrite incorrect and told them that ive noticed my bot count go down drastically... they then told me that they have blocked some bots, however they told me that they left the major good bots in place.

    Ive asked them several times when this change took place, but they were not able to tell me several times... (thus assuming they must of done this long time ago and cannot remember...) i wanted to see whether it coincides with when i installed vbseo... as if it dosent then its not a server side issue, but vbseo taking its time? I didnt know whether it was vbseo or the host at fault.. im just hoping its not a change they made recently.

    Ive asked them to tell me what bots are on the allowed list and i will also ask them to give me a list of those that they have denied and possibly get back to you with the result.


    Cheers

  8. #8
    Senior Member Array
    Real Name
    Joseph Ward
    Join Date
    Jun 2005
    Posts
    23,845
    Liked
    47 times
    Blog Entries
    9
    Yes - definitely ask them to give you the full list of disallowed bots. Ask them for ballpark estimates for when that was setup. They should be able to give you any idea.

    If they do not give you a list, I would have a hard time trusting them. They should be able to provide you with that.

    Which host are you using?

  9. #9
    Senior Member Array AdamFL's Avatar
    Real Name
    Adam
    Join Date
    Oct 2008
    Location
    South Florida
    Posts
    167
    Liked
    0 times
    I was wondering if i can use the exact robots.txt provided on the link posted earlier for my forums:
    http://toomanysecrets.com/robots.txt
    thank you

  10. #10
    Senior Member Array
    Real Name
    Brian
    Join Date
    Apr 2006
    Posts
    6,983
    Liked
    10 times
    Well, chances are you don't have the same directories, so i would say no.


    IMO, unless your breaking your bandwidth bill, or your site is incredibly slow due to huge bot usage, you shouldn't disallow any particular robot.

    I've outlined a suggested robots file in my Ultimate Guide.

Similar Threads

  1. Reduce Bandwidth, Enhance Pagerank [Remove Last Post for Guests]
    By dutchbb in forum Template Modifications
    Replies: 68
    Last Post: 08-03-2011, 01:27 AM
  2. New Kid On The Block
    By flixray in forum Introduce Yourself
    Replies: 5
    Last Post: 01-06-2008, 03:57 PM
  3. Bandwidth consumption
    By Sola in forum General Discussion
    Replies: 6
    Last Post: 07-24-2007, 06:10 PM
  4. Trying to block some bandwidth leeches
    By FightRice in forum General Discussion
    Replies: 3
    Last Post: 09-16-2006, 04:40 PM

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •