LinkBack URL
About LinkBacksHello,
Is it possible to limit the yahoo spiders crawling my web site? I'm getting 400-500 yahoo spiders (online almost all day long) per day and i can't understand why it's taking so long... My servers aren't overloaded and they are serving content really fast especially for US visitors.
Indexed pages for msxlabs.org:
Google: 1,160,000
Yahoo: 493,916
Live: 3.140
Results 1 to 3 of 3
Hundreds of Yahoo! Spiders
This is a discussion on Hundreds of Yahoo! Spiders within the General Discussion forums, part of the vBulletin SEO Discussion category; Hello, Is it possible to limit the yahoo spiders crawling my web site? I'm getting 400-500 yahoo spiders (online almost ...
-
04-07-2008 07:37 PM #1Senior Member
- Real Name
- Mavi KARANLIK
- Join Date
- Feb 2006
- Location
- Ankara/TÜRKİYE
- Posts
- 310
- Liked
- 1 times
Hundreds of Yahoo! Spiders
-
04-07-2008 07:42 PM #2Senior Member
- Real Name
- Mavi KARANLIK
- Join Date
- Feb 2006
- Location
- Ankara/TÜRKİYE
- Posts
- 310
- Liked
- 1 times
-
04-07-2008 08:06 PM #3Senior Member
- Real Name
- Mavi KARANLIK
- Join Date
- Feb 2006
- Location
- Ankara/TÜRKİYE
- Posts
- 310
- Liked
- 1 times
Updated the robots.txt file 10 minutes ago with the delay code and i have 150 yahoo bots instead of 410 (and getting smaller)
Reply With QuoteSimilar Threads
-
Spiders!
By sohan in forum Analysis: Traffic & SERPSReplies: 5Last Post: 11-11-2006, 07:40 PM -
Hundreds of yahoo slurp robots when vbseo is disabled
By mcyates in forum General DiscussionReplies: 2Last Post: 10-30-2006, 02:26 PM -
How Many Spiders...?
By Atheda in forum General DiscussionReplies: 5Last Post: 05-18-2006, 05:44 PM -
increasing yahoo slurp spiders
By MentaL in forum General DiscussionReplies: 0Last Post: 01-20-2006, 01:08 AM -
Yahoo only spiders external.php
By caliman in forum General DiscussionReplies: 2Last Post: 12-03-2005, 04:42 PM
