SEO

Hi all

Can anyone tell me how to restrict the Yahoo! Slurp spider.

Google and MSN Bots are fine, they come, crawl and leave.
Yahoo! Slurp however is a constant presence and will not go away.
Can be up to 20 listed in Who's online 24/7 .

vBulletin 3.5.4 - vBSEO 2.4.0

Hello Mal:

You can add this to your robots.txt to block Yahoo:

User-agent: Slurp
Disallow: /

__________________
My Blog | GPS Discussion Forum

Hi Keith

I'd rather not block it completely, just restrict it .

Did you tried rel="nofollow" on your link? It should stop any crawler to follow that specific link.

The problem is that that they just sit there on the index page .

The robots.text entry has blocked them for now, however I don't see this as the long term answer.
The others come and go, what is it with Slurp :(
It's like they have decided to squat . lol

I would suggest just letting them do their thing. If they are just sitting there, they aren't using your bandwidth, so you shouldn't have anything to worry about.

__________________
My Blog | GPS Discussion Forum

Just an update on this.

slurp is still camped out on my site. Have had some advice saying it may well be related to the RSS/XML feed.

Anyway, having received a few "too many connections" errors, I added this to my robots.txt file


No errors since.

I've upped my crawl delay to 60, Yahoo need to sort out their crawler, it's going nuts on my site.