Results 1 to 5 of 5

Bing Bots

This is a discussion on Bing Bots within the General Discussion forums, part of the vBSEO Google/Yahoo Sitemap category; Looking for advice on how to reduce the number of Bing bots at visit my forum at one time. It's ...

  1. #1
    Member alexscott's Avatar
    Real Name
    Alex
    Join Date
    Jul 2009
    Posts
    39
    Liked
    0 times

    Bing Bots

    Looking for advice on how to reduce the number of Bing bots at visit my forum at one time.
    It's usually over 10 at any time, were Google etc only ever have one crawling.
    I've tried using their webmaster site but that didn't have any effect and also using the htaccess.

    Sure I read somewhere that this could slow things down?

  2. #2
    vBSEO.com Webmaster Mert Gökçeimam's Avatar
    Real Name
    Lizard King
    Join Date
    Oct 2005
    Location
    Istanbul, Turkey, Turkey
    Posts
    23,100
    Liked
    622 times
    Blog Entries
    4
    Hello ,

    You can delay them via robots.txt file
    Mert Gökçeimam / Crawlability Inc.

    vBSEO 3.6.0 Alpha Önizlemesi - Including Like Tree
    Unveiling the NEW vBSEO Sitemap Generator 3.0 - available NOW for vBSEO Customers!


    Twitter:@Depkac
    Personal Blog : Mert Gökçeimam

  3. #3
    Member alexscott's Avatar
    Real Name
    Alex
    Join Date
    Jul 2009
    Posts
    39
    Liked
    0 times
    Mert, sorry. Meant robots.txt file instead of the htaccess.
    I have this in it,

    User-agent: Bing
    Crawl-delay: 60

    Will this make a difference, hasn't so far or do you recommend something else.

  4. #4
    Junior Member
    Real Name
    Marco Polo
    Join Date
    Dec 2011
    Location
    Pepe's Construction Jard
    Posts
    4
    Liked
    0 times
    Any idea as to why the Bing bot behaves in this way compared to the Google bot? I would assume that it is just consuming a site's bandwidth by crawling a website with a lot of bots at a given time.

    However, limit the Bing Bot on your robots.txt file, and this should do the trick.

  5. #5
    vBSEO Staff Brian Cummiskey's Avatar
    Real Name
    Brian Cummiskey
    Join Date
    Jul 2009
    Location
    btwn NYC and Boston
    Posts
    12,789
    Liked
    657 times
    Blog Entries
    2
    crawl-delay is the only setting available, and even that is merely a suggestion. Ultimately, the bots will do what they want to do. If they are bringin down your server, perhaps you should consider adding more server resources as even 100+ extra anything shouldn't bring you down. What happens if someone drops a link to a thread of yours on reddit, or some other super popular site? You don't want your server to crash and lose out on all that traffic.

Similar Threads

  1. Bing.com
    By Mert Gökçeimam in forum General Discussion
    Replies: 45
    Last Post: 11-11-2009, 02:27 PM
  2. Bing
    By Invictus in forum Sitemap Features Archive
    Replies: 15
    Last Post: 08-04-2009, 07:30 AM
  3. Sitemap for Bing
    By DADOCTOR in forum General Discussion
    Replies: 1
    Last Post: 06-26-2009, 05:38 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •