vBulletin SEO Forums

SEO

vBulletin Search Engine Optimization

Buy vBSEO Now! HACKER SAFE certified sites prevent over 99.9% of hacker crime.
ne nw
vBSEO Total Support Team Launches DeskPro New vBSEO Discount Level for Network Builders vBSEO 3.2.0 GOLD Has Landed Success with vBSEO = 600ore Web Visitors + $1400 in a Day! Crawlability Inc. Files for SEO Technology Patent
se sw

Yahoo Bot Crawl Delay: Any harm?

This is a discussion on Yahoo Bot Crawl Delay: Any harm? within the General Discussion forums, part of the vBulletin SEO Discussion category; Hello, we still have ~400 Yahoo Bots online at our site, which generally spoken is a good thing. Even with ...

Go Back   vBulletin SEO Forums > vBulletin SEO Discussion > General Discussion

Yahoo Bot Crawl Delay: Any harm? Yahoo Bot Crawl Delay: Any harm?

Enhancing 80 million pages.

 Register FAQ Members List Social Groups Calendar Search Today's Posts Mark Forums Read

Reply

 

LinkBack Thread Tools
  #1  
Old 08-07-2006, 03:23 AM
Lian's Avatar
Senior Member
Big Board Administrator
  
Real Name: Lian
Join Date: Jun 2006
Location: Munich, DE
Posts: 293
Question Yahoo Bot Crawl Delay: Any harm?
Hello,

we still have ~400 Yahoo Bots online at our site, which generally spoken is a good thing.
Even with their new Bot recently relaunched we have that many bots online.

But the performance impact is noticable, so we think of placing a crawl delay for the bot in our robots.txt, see:
Yahoo! Help -
We think of
Code:
User-agent: Slurp
Crawl-delay: 5
Will that do any harm SEO-wise?

Will it help reducing the queries of Y Bot to reduce our server's load?
I guess so...

Thanks,
Lian
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #2  
Old 08-07-2006, 05:44 AM
Lian's Avatar
Senior Member
Big Board Administrator
  
Real Name: Lian
Join Date: Jun 2006
Location: Munich, DE
Posts: 293
Found it:
Quote:
Setting a crawl-delay may limit the coverage and freshness of your content representation in Yahoo! search results. If you do feel that a crawl-delay is necessary, use small values to avoid blocking Slurp discovery and refresh of your key content.
Nope, we surely don't want that.
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #3  
Old 08-07-2006, 01:50 PM
Joe Ward's Avatar
vBSEO Staff
vBSEO Total Customer SupportvBSEO Documenter
  
Real Name: Joseph Ward
Join Date: Jun 2005
Location: Puerto Rico
Posts: 19,748
Blog Entries: 7
FYI Yahoo released a new version of Slurp. It's supported to be more efficient in crawling and aims to reduce bandwidth consumption (and performance impact):

Yahoo Slurp - New Version of Search Engine Crawler Released
__________________
Joe Ward / Crawlability Inc.
Support Team Launches New DeskPro Powered Tool Enhanced Support at Your Service

vBSEO 3.2.0 Launched - Maximum Overdrive for Your Web Traffic! Over 100 Instant SEO Optimizations

6X Traffic - $1400 in One Day with vBSEO! Imagine What the vBSEO Patent Pending Technology Can Do For You.
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #4  
Old 08-07-2006, 03:50 PM
Lian's Avatar
Senior Member
Big Board Administrator
  
Real Name: Lian
Join Date: Jun 2006
Location: Munich, DE
Posts: 293
Thanks, I've read the thread.

I wonder why we still have ~400 online during (bot) peak times...
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
Reply

« SEO all skins? or only the main one | Changing name of forum directory »
Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On

Similar Threads

Thread Thread Starter Forum Replies Last Post
Yahoo Slurp - New Version of Search Engine Crawler Released Joe Ward SEO Buzz 4 08-16-2006 04:35 AM


All times are GMT -4. The time now is 12:11 AM.

Contact Us - vBSEO Home - Site Map - Top

Powered by vBulletin Version 3.8.0 Beta 4
Copyright ©2000 - 2008, Jelsoft Enterprises Ltd.
SEO by vBSEO 3.2.5 ©2008, Crawlability, Inc.