vBulletin 4, the most powerful community software + vBSEO 3.5, the ultimate SEO solution = Your ultimate platform for 2010 and beyond. Click below to learn more.

Results 1 to 12 of 12

Google Webmaster tools- crawl rate

This is a discussion on Google Webmaster tools- crawl rate within the General Discussion forums, part of the vBulletin SEO Discussion category; I have just started using Google Webmaster tools...I stumbled upon "Set crawl rate." It seemed to make sense to set ...

  1. #1
    Member JADM's Avatar
    Real Name
    Jason
    Join Date
    Dec 2007
    Posts
    72
    Blog Entries
    18

    Google Webmaster tools- crawl rate

    I have just started using Google Webmaster tools...I stumbled upon "Set crawl rate." It seemed to make sense to set to faster crawl. Any downside (other than server load) to this? I assume the server load effect is not that great on a smaller forum like mine. Anything I am missing here?

  2. #2
    Senior Member briansol's Avatar
    Real Name
    Brian
    Join Date
    Apr 2006
    Location
    Central CT, USA
    Posts
    6,983
    usually you have medium and slow as options, not fast.

    unless you're doing 5000 posts a day, i would leave it at medium.

  3. #3
    Member JADM's Avatar
    Real Name
    Jason
    Join Date
    Dec 2007
    Posts
    72
    Blog Entries
    18
    briansol,

    Thanks for the reply. I am looking at Dashboard >> tools >> Set crawl rate. It says "We've detected that Googlebot is limiting the rate at which it crawls pages on your site to ensure it doesn't use too much of your server's resources. If your server can handle additional Googlebot traffic, we recommend that you choose Faster below." I am WAYYYY below 5,000 post/day . But is there a reason NOT to do this? Or do you think it just doesn't matter?

  4. #4
    Senior Member briansol's Avatar
    Real Name
    Brian
    Join Date
    Apr 2006
    Location
    Central CT, USA
    Posts
    6,983
    it will just eat more processor on your server. I see no reason to change it from normal. Ive had mine there since day 1, and the faster option isn't even available to me.

    Faster At this time, crawl rate is not a factor in your site's crawl.

  5. #5
    Junior Member
    Real Name
    SEO Meter
    Join Date
    Jan 2008
    Posts
    3
    If you choose to use that option, you have to re-set the crawl rate once every 90 days.
    Otherwise, the crawl rate will be back to its default rate.

    More about on Google's crawl rate:
    What Determines Search Engine’s Crawl Rate? - Search Engine Optimization Blog

  6. #6
    Member JADM's Avatar
    Real Name
    Jason
    Join Date
    Dec 2007
    Posts
    72
    Blog Entries
    18
    I'll take a look at the blog, but do you have an idea of why this might be important. If it is the same info sent the same day, I would think this is negligible/not important. But then, I am stumped on why they would offer this option if it has no signifigance.

  7. #7
    Senior Member
    Real Name
    Dhillon
    Join Date
    Apr 2006
    Posts
    327
    When I enabled fast option, googlebot hit rate increased by 15-20%.
    If your server can handle that much load,Enable it

  8. #8
    Senior Member
    Real Name
    A.D
    Join Date
    Oct 2007
    Location
    France
    Posts
    159
    Quote Originally Posted by seometer View Post
    If you choose to use that option, you have to re-set the crawl rate once every 90 days.
    Otherwise, the crawl rate will be back to its default rate.

    More about on Google's crawl rate:
    What Determines Search Engine’s Crawl Rate? - Search Engine Optimization Blog
    Hey good job on your site SEOmaster and thanks for the add

  9. #9
    Member JADM's Avatar
    Real Name
    Jason
    Join Date
    Dec 2007
    Posts
    72
    Blog Entries
    18
    I am on shared hosting at GoDaddy...not really sure how to tell if it can handle it....so, I am going to set it for fast crawl and see what happens! (Probably not the preferred method, but hey, nothing ventured, nothing gained)!

  10. #10
    Senior Member
    Real Name
    A.D
    Join Date
    Oct 2007
    Location
    France
    Posts
    159
    no you can JADM, even with a fast google crawl rate , I see far much bots from msn live or yahoo.

  11. #11
    Junior Member
    Real Name
    SEO Meter
    Join Date
    Jan 2008
    Posts
    3
    Quote Originally Posted by JADM View Post
    I am stumped on why they would offer this option if it has no signifigance.
    I guess they wanted to have some sort of human input on their crawling algo, esp in case crawling rate got unduly influenced by temporary screwy hosting setup.

  12. #12
    Junior Member
    Real Name
    SEO Meter
    Join Date
    Jan 2008
    Posts
    3
    Quote Originally Posted by class101 View Post
    Hey good job on your site SEOmaster and thanks for the add
    Thanks. My pleasure.

Similar Threads

  1. Replies: 1
    Last Post: 08-22-2007, 06:15 AM
  2. Google Webmaster Tools
    By OnR in forum Türkçe
    Replies: 2
    Last Post: 07-12-2007, 12:42 PM
  3. Weird entry in Google webmaster tools.
    By Dave Hybrid in forum General Discussion
    Replies: 5
    Last Post: 05-30-2007, 11:45 AM
  4. Replies: 0
    Last Post: 04-21-2007, 06:30 AM
  5. Google Webmaster Tools - Matt Cutts Video
    By Joe Ward in forum General Discussion
    Replies: 2
    Last Post: 11-01-2006, 02:33 AM

Tags for this Thread