vBulletin SEO Forums

SEO

vBulletin Search Engine Optimization

Buy vBSEO Now!
vBSEO 2.0 Style Released vBSEO 3.3.0 GOLD Launched vBSEO's "LiveStats" for Google Analytics vB Sitemap Generator, Version 2.5 Success with vBSEO = 600ore Web Visitors + $1400 in a Day!

Google Webmaster tools- crawl rate

This is a discussion on Google Webmaster tools- crawl rate within the General Discussion forums, part of the vBulletin SEO Discussion category; I have just started using Google Webmaster tools...I stumbled upon "Set crawl rate." It seemed to make sense to set ...

Go Back   vBulletin SEO Forums > vBulletin SEO Discussion > General Discussion

Enhancing 80 million pages.

  #1  
Old 01-09-2008, 11:30 PM
JADM's Avatar
Member
 
Real Name: Jason
Join Date: Dec 2007
Posts: 72
Blog Entries: 18
Google Webmaster tools- crawl rate

I have just started using Google Webmaster tools...I stumbled upon "Set crawl rate." It seemed to make sense to set to faster crawl. Any downside (other than server load) to this? I assume the server load effect is not that great on a smaller forum like mine. Anything I am missing here?
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #2  
Old 01-09-2008, 11:58 PM
briansol's Avatar
Senior Member
 
Real Name: Brian
Join Date: Apr 2006
Location: Central CT, USA
Posts: 7,090
usually you have medium and slow as options, not fast.

unless you're doing 5000 posts a day, i would leave it at medium.
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #3  
Old 01-10-2008, 12:17 AM
JADM's Avatar
Member
 
Real Name: Jason
Join Date: Dec 2007
Posts: 72
Blog Entries: 18
briansol,

Thanks for the reply. I am looking at Dashboard >> tools >> Set crawl rate. It says "We've detected that Googlebot is limiting the rate at which it crawls pages on your site to ensure it doesn't use too much of your server's resources. If your server can handle additional Googlebot traffic, we recommend that you choose Faster below." I am WAYYYY below 5,000 post/day . But is there a reason NOT to do this? Or do you think it just doesn't matter?
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #4  
Old 01-10-2008, 12:28 AM
briansol's Avatar
Senior Member
 
Real Name: Brian
Join Date: Apr 2006
Location: Central CT, USA
Posts: 7,090
it will just eat more processor on your server. I see no reason to change it from normal. Ive had mine there since day 1, and the faster option isn't even available to me.

Faster At this time, crawl rate is not a factor in your site's crawl.
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #5  
Old 01-10-2008, 04:43 PM
Junior Member
 
Real Name: SEO Meter
Join Date: Jan 2008
Posts: 3
If you choose to use that option, you have to re-set the crawl rate once every 90 days.
Otherwise, the crawl rate will be back to its default rate.

More about on Google's crawl rate:
What Determines Search Engine’s Crawl Rate? - Search Engine Optimization Blog
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #6  
Old 01-10-2008, 05:18 PM
JADM's Avatar
Member
 
Real Name: Jason
Join Date: Dec 2007
Posts: 72
Blog Entries: 18
I'll take a look at the blog, but do you have an idea of why this might be important. If it is the same info sent the same day, I would think this is negligible/not important. But then, I am stumped on why they would offer this option if it has no signifigance.
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #7  
Old 01-10-2008, 05:41 PM
Senior Member
Big Board Administrator
 
Real Name: Dhillon
Join Date: Apr 2006
Posts: 311
When I enabled fast option, googlebot hit rate increased by 15-20%.
If your server can handle that much load,Enable it
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #8  
Old 01-11-2008, 07:19 AM
Senior Member
 
Real Name: A.D
Join Date: Oct 2007
Location: France
Posts: 158
Quote:
Originally Posted by seometer View Post
If you choose to use that option, you have to re-set the crawl rate once every 90 days.
Otherwise, the crawl rate will be back to its default rate.

More about on Google's crawl rate:
What Determines Search Engine’s Crawl Rate? - Search Engine Optimization Blog
Hey good job on your site SEOmaster and thanks for the add
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #9  
Old 01-11-2008, 11:52 AM
JADM's Avatar
Member
 
Real Name: Jason
Join Date: Dec 2007
Posts: 72
Blog Entries: 18
I am on shared hosting at GoDaddy...not really sure how to tell if it can handle it....so, I am going to set it for fast crawl and see what happens! (Probably not the preferred method, but hey, nothing ventured, nothing gained)!
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #10  
Old 01-11-2008, 02:07 PM
Senior Member
 
Real Name: A.D
Join Date: Oct 2007
Location: France
Posts: 158
no you can JADM, even with a fast google crawl rate , I see far much bots from msn live or yahoo.
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #11  
Old 01-11-2008, 05:26 PM
Junior Member
 
Real Name: SEO Meter
Join Date: Jan 2008
Posts: 3
Quote:
Originally Posted by JADM View Post
I am stumped on why they would offer this option if it has no signifigance.
I guess they wanted to have some sort of human input on their crawling algo, esp in case crawling rate got unduly influenced by temporary screwy hosting setup.
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #12  
Old 01-11-2008, 05:26 PM
Junior Member
 
Real Name: SEO Meter
Join Date: Jan 2008
Posts: 3
Quote:
Originally Posted by class101 View Post
Hey good job on your site SEOmaster and thanks for the add
Thanks. My pleasure.
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
Reply

Tags
bots, crawl rate, google webmaster tools

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads

Thread Thread Starter Forum Replies Last Post
Google webmaster tools - Network Uncreachable OnR Türkçe 1 08-22-2007 05:15 AM
Google Webmaster Tools OnR Türkçe 2 07-12-2007 11:42 AM
Weird entry in Google webmaster tools. Dave Hybrid General Discussion 5 05-30-2007 10:45 AM
New SEO Tools: Robots.txt Builder, Term Targeting & Crawl Test Eros SEO Buzz 0 04-21-2007 05:30 AM
Google Webmaster Tools - Matt Cutts Video Joe Ward SEO Buzz 2 11-01-2006 01:33 AM


All times are GMT -4. The time now is 03:49 PM.


Powered by vBulletin Version 3.8.3
Copyright ©2000 - 2009, Jelsoft Enterprises Ltd.
SEO by vBSEO 3.3.0 ©2009, Crawlability, Inc.