200+ Spiders, same IP, all looking at the same page...

What is this all about? They are all looking at the same page, accessed at the same time and all have the same IP. This doesn't seem right. Can anyone explain this please?

Many thanks.

This is a one time occurrence or has happened repeatedly?

slurp has been retarded lately.

i suggest adding

Code:

User-agent: Slurp
Crawl-delay: 60

to your robots.txt file

Originally Posted by Joe Ward

This is a one time occurrence or has happened repeatedly?

Not sure, I'll look into this. They are not doing it at this moment.

Also Slurp is looking at User Profiles even though robots.txt says not to and vBSEO is set to add no follow tags to links to profiles. Strange...

Originally Posted by briansol

slurp has been retarded lately.

i suggest adding

Code:

User-agent: Slurp
Crawl-delay: 60

to your robots.txt file

I already have this in there:

Code:

User-agent: Slurp
Crawl-delay: 120

It is doing it again. I check just now and there are 150 slurps on my site, all with the same ip, all the same access time and all look at User Profile - don't know which, but probably the same one. Also, as above, Slurp shouldn't be looking at profiles anyway.

I've not noticed this at any other time of day, but I'll keep an eye out. I've been getting load issues in the morning, I wonder if this is why? (I'm in the UK btw).

Just looked in more detail, and it is the same user profile in the Photopost Gallery it is looking at, not the vBulletin profile page. :( So each time, it is photopost related pages causing this.

i'm having same problem but with 2k yahoo spiders hehehe :( look my forums Under-Linux.Org Fóruns

Seems this is still happening. It only happens in photopost and only with slurps - I think somehow they are getting stuck in there or something. How can I stop slurps going into the gallery entirely? (../gallery/)

Thanks.

Mike, do you have a robots.txt file? Assuming your robots.txt file is in / and gallery is in /gallery, you can try the following in robots.txt file:

Code:

User-agent: Slurp
Disallow: /gallery