yahoo slurp

Hi all

Can anyone tell me how to restrict the Yahoo! Slurp spider.

Google and MSN Bots are fine, they come, crawl and leave.
Yahoo! Slurp however is a constant presence and will not go away.
Can be up to 20 listed in Who's online 24/7 .

vBulletin 3.5.4 - vBSEO 2.4.0

Hello Mal:

You can add this to your robots.txt to block Yahoo:

User-agent: Slurp
Disallow: /

Hi Keith

I'd rather not block it completely, just restrict it .

Did you tried rel="nofollow" on your link? It should stop any crawler to follow that specific link.

The problem is that that they just sit there on the index page .

The robots.text entry has blocked them for now, however I don't see this as the long term answer.
The others come and go, what is it with Slurp :(
It's like they have decided to squat . lol

I would suggest just letting them do their thing. If they are just sitting there, they aren't using your bandwidth, so you shouldn't have anything to worry about.

Just an update on this.

slurp is still camped out on my site. Have had some advice saying it may well be related to the RSS/XML feed.

Anyway, having received a few "too many connections" errors, I added this to my robots.txt file

Code:

 
User-agent: Slurp 
Crawl-delay: 5

No errors since.

I've upped my crawl delay to 60, Yahoo need to sort out their crawler, it's going nuts on my site.