vBulletin SEO Forums

SEO

vBulletin Search Engine Optimization

Buy vBSEO Now! HACKER SAFE certified sites prevent over 99.9% of hacker crime.
ne nw
vBSEO Total Support Team Launches DeskPro New vBSEO Discount Level for Network Builders vBSEO 3.2.0 GOLD Has Landed Success with vBSEO = 600ore Web Visitors + $1400 in a Day! Crawlability Inc. Files for SEO Technology Patent
se sw

yahoo slurp

This is a discussion on yahoo slurp within the General Discussion forums, part of the vBulletin SEO Discussion category; Hi all Can anyone tell me how to restrict the Yahoo! Slurp spider. Google and MSN Bots are fine, they ...

Go Back   vBulletin SEO Forums > vBulletin SEO Discussion > General Discussion

yahoo slurp yahoo slurp

Enhancing 80 million pages.

 Register FAQ Members List Social Groups Calendar Search Today's Posts Mark Forums Read

Reply

 

LinkBack Thread Tools
  #1  
Old 04-20-2006, 10:38 PM
Member
  
Real Name: mal
Join Date: Mar 2006
Posts: 58
Question yahoo slurp
Hi all

Can anyone tell me how to restrict the Yahoo! Slurp spider.

Google and MSN Bots are fine, they come, crawl and leave.
Yahoo! Slurp however is a constant presence and will not go away.
Can be up to 20 listed in Who's online 24/7 .

vBulletin 3.5.4 - vBSEO 2.4.0
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #2  
Old 04-20-2006, 10:39 PM
Keith Cohen's Avatar
vBSEO Staff
vBSEO Total Customer SupportBig Board Administrator
  
Real Name: Keith Cohen
Join Date: Jul 2005
Location: Raleigh, NC USA
Posts: 6,266
Re: yahoo slurp
Hello Mal:

You can add this to your robots.txt to block Yahoo:

User-agent: Slurp
Disallow: /
__________________
Keith Cohen / Crawlability Inc.
Support Team Launches New DeskPro Powered Tool Enhanced Support at Your Service

vBSEO 3.2.0 Launched - Maximum Overdrive for Your Web Traffic! Over 100 Instant SEO Optimizations

6X Traffic - $1400 in One Day with vBSEO! Imagine What the vBSEO Patent Pending Technology Can Do For You.


My Personal Sites: My Blog | GPS Discussion Forum
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #3  
Old 04-20-2006, 10:47 PM
Member
  
Real Name: mal
Join Date: Mar 2006
Posts: 58
Re: yahoo slurp
Hi Keith

I'd rather not block it completely, just restrict it .
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #4  
Old 04-21-2006, 12:24 PM
TECK's Avatar
Senior Member
vBulletin HackervBSEO Pre-Release TeamDesign for SEOvBSEO Code Quality Assurance
  
Real Name: Floren Munteanu
Join Date: Apr 2006
Location: Canada
Posts: 616
Blog Entries: 1
Re: yahoo slurp
Did you tried rel="nofollow" on your link? It should stop any crawler to follow that specific link.
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #5  
Old 04-22-2006, 02:03 PM
Member
  
Real Name: mal
Join Date: Mar 2006
Posts: 58
Re: yahoo slurp
The problem is that that they just sit there on the index page .

The robots.text entry has blocked them for now, however I don't see this as the long term answer.
The others come and go, what is it with Slurp :(
It's like they have decided to squat . lol
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #6  
Old 04-22-2006, 02:22 PM
Keith Cohen's Avatar
vBSEO Staff
vBSEO Total Customer SupportBig Board Administrator
  
Real Name: Keith Cohen
Join Date: Jul 2005
Location: Raleigh, NC USA
Posts: 6,266
Re: yahoo slurp
I would suggest just letting them do their thing. If they are just sitting there, they aren't using your bandwidth, so you shouldn't have anything to worry about.
__________________
Keith Cohen / Crawlability Inc.
Support Team Launches New DeskPro Powered Tool Enhanced Support at Your Service

vBSEO 3.2.0 Launched - Maximum Overdrive for Your Web Traffic! Over 100 Instant SEO Optimizations

6X Traffic - $1400 in One Day with vBSEO! Imagine What the vBSEO Patent Pending Technology Can Do For You.


My Personal Sites: My Blog | GPS Discussion Forum
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #7  
Old 05-19-2006, 06:23 PM
Member
  
Real Name: mal
Join Date: Mar 2006
Posts: 58
Re: yahoo slurp
Just an update on this.

slurp is still camped out on my site. Have had some advice saying it may well be related to the RSS/XML feed.

Anyway, having received a few "too many connections" errors, I added this to my robots.txt file


Code:
 
User-agent: Slurp 
Crawl-delay: 5
No errors since.
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #8  
Old 07-21-2007, 11:22 PM
Junior Member
  
Join Date: Nov 2005
Posts: 26
I've upped my crawl delay to 60, Yahoo need to sort out their crawler, it's going nuts on my site.
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
Reply

« SEO Help | Getting PR back... how long does it take? »
Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On

Similar Threads

Thread Thread Starter Forum Replies Last Post
Yahoo doesnt like my site... Barry General Discussion 6 02-04-2006 03:24 AM
increasing yahoo slurp spiders MentaL General Discussion 0 01-20-2006 01:08 AM
Why does Yahoo hate me? caliman Off-Topic & Chit Chat 5 11-18-2005 03:31 AM
Yahoo not reconigzing the vbseo urls emin General Discussion 5 11-14-2005 09:05 PM
VBSEO and YAHOO Slurp MentaL General Discussion 6 11-02-2005 11:11 AM


All times are GMT -4. The time now is 12:15 AM.

Contact Us - vBSEO Home - Site Map - Top

Powered by vBulletin Version 3.8.0 Beta 4
Copyright ©2000 - 2008, Jelsoft Enterprises Ltd.
SEO by vBSEO 3.2.5 ©2008, Crawlability, Inc.