vBulletin SEO Forums

SEO

vBulletin Search Engine Optimization

Buy vBSEO Now! HACKER SAFE certified sites prevent over 99.9% of hacker crime.
ne nw
vBSEO Total Support Team Launches DeskPro New vBSEO Discount Level for Network Builders vBSEO 3.2.0 GOLD Has Landed Success with vBSEO = 600ore Web Visitors + $1400 in a Day! Crawlability Inc. Files for SEO Technology Patent
se sw

200+ Spiders, same IP, all looking at the same page...

This is a discussion on 200+ Spiders, same IP, all looking at the same page... within the General Discussion forums, part of the vBulletin SEO Discussion category; What is this all about? They are all looking at the same page, accessed at the same time and all ...

Go Back   vBulletin SEO Forums > vBulletin SEO Discussion > General Discussion

200+ Spiders, same IP, all looking at the same page... 200+ Spiders, same IP, all looking at the same page...

Enhancing 80 million pages.

 Register FAQ Members List Social Groups Calendar Search Today's Posts Mark Forums Read

Reply

 

LinkBack Thread Tools
  #1  
Old 06-09-2008, 04:23 AM
Member
  
Real Name: Mike Warner
Join Date: Feb 2006
Posts: 51
200+ Spiders, same IP, all looking at the same page...
What is this all about? They are all looking at the same page, accessed at the same time and all have the same IP. This doesn't seem right. Can anyone explain this please?

Many thanks.
Attached Thumbnails
spiders2.gif  
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #2  
Old 06-09-2008, 12:31 PM
Joe Ward's Avatar
vBSEO Staff
vBSEO Total Customer SupportvBSEO Documenter
  
Real Name: Joseph Ward
Join Date: Jun 2005
Location: Puerto Rico
Posts: 19,749
Blog Entries: 7
This is a one time occurrence or has happened repeatedly?
__________________
Joe Ward / Crawlability Inc.
Support Team Launches New DeskPro Powered Tool Enhanced Support at Your Service

vBSEO 3.2.0 Launched - Maximum Overdrive for Your Web Traffic! Over 100 Instant SEO Optimizations

6X Traffic - $1400 in One Day with vBSEO! Imagine What the vBSEO Patent Pending Technology Can Do For You.
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #3  
Old 06-09-2008, 01:12 PM
briansol's Avatar
Senior Member
vBSEO Pre-Release TeamDesign for SEOBig Board Administrator
  
Real Name: Brian
Join Date: Apr 2006
Location: Central CT, USA
Posts: 5,545
slurp has been retarded lately.

i suggest adding

Code:
User-agent: Slurp
Crawl-delay: 60
to your robots.txt file
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #4  
Old 06-10-2008, 08:57 AM
Member
  
Real Name: Mike Warner
Join Date: Feb 2006
Posts: 51
Thumbs up
Quote:
Originally Posted by Joe Ward View Post
This is a one time occurrence or has happened repeatedly?
Not sure, I'll look into this. They are not doing it at this moment.

Also Slurp is looking at User Profiles even though robots.txt says not to and vBSEO is set to add no follow tags to links to profiles. Strange...
Quote:
Originally Posted by briansol View Post
slurp has been retarded lately.

i suggest adding

Code:
User-agent: Slurp
Crawl-delay: 60
to your robots.txt file
I already have this in there:
Code:
User-agent: Slurp
Crawl-delay: 120
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #5  
Old 06-11-2008, 04:38 AM
Member
  
Real Name: Mike Warner
Join Date: Feb 2006
Posts: 51
It is doing it again. I check just now and there are 150 slurps on my site, all with the same ip, all the same access time and all look at User Profile - don't know which, but probably the same one. Also, as above, Slurp shouldn't be looking at profiles anyway.

I've not noticed this at any other time of day, but I'll keep an eye out. I've been getting load issues in the morning, I wonder if this is why? (I'm in the UK btw).
Attached Thumbnails
spiders3.gif  
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #6  
Old 06-11-2008, 04:44 AM
Member
  
Real Name: Mike Warner
Join Date: Feb 2006
Posts: 51
Just looked in more detail, and it is the same user profile in the Photopost Gallery it is looking at, not the vBulletin profile page. :( So each time, it is photopost related pages causing this.
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #7  
Old 06-12-2008, 01:33 AM
Junior Member
  
Real Name: Marcus Maciel
Join Date: Oct 2006
Location: Brazil
Posts: 19
i'm having same problem but with 2k yahoo spiders hehehe :( look my forums Under-Linux.Org Fóruns
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #8  
Old 06-19-2008, 03:49 AM
Member
  
Real Name: Mike Warner
Join Date: Feb 2006
Posts: 51
Seems this is still happening. It only happens in photopost and only with slurps - I think somehow they are getting stuck in there or something. How can I stop slurps going into the gallery entirely? (../gallery/)

Thanks.
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #9  
Old 06-19-2008, 04:02 PM
vBSEO Staff
vBSEO Total Customer Support
  
Real Name: Asad Khan
Join Date: Jun 2008
Posts: 229
Mike, do you have a robots.txt file? Assuming your robots.txt file is in / and gallery is in /gallery, you can try the following in robots.txt file:

Code:
User-agent: Slurp
Disallow: /gallery
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
Reply

« RSS Feeds good or bad for seo | Add www to forum url with no trailing slash »
Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On

Similar Threads

Thread Thread Starter Forum Replies Last Post
How do spiders Checks ur page? MissKalunji General Discussion 6 03-05-2008 01:46 PM
where are the spiders? obx11 General Discussion 8 03-04-2007 12:17 AM
Spiders! sohan vBSEO Reviews 5 11-11-2006 07:40 PM
How Many Spiders...? Paul Govier General Discussion 5 05-18-2006 05:44 PM


All times are GMT -4. The time now is 01:16 AM.

Contact Us - vBSEO Home - Site Map - Top

Powered by vBulletin Version 3.8.0 Beta 4
Copyright ©2000 - 2008, Jelsoft Enterprises Ltd.
SEO by vBSEO 3.2.5 ©2008, Crawlability, Inc.