SEO
vBulletin Search Engine Optimization
Hundreds of Yahoo! Spiders
This is a discussion on Hundreds of Yahoo! Spiders within the General Discussion forums, part of the vBulletin SEO Discussion category; Hello, Is it possible to limit the yahoo spiders crawling my web site? I'm getting 400-500 yahoo spiders (online almost ...
| |||||||
Enhancing 80 million pages. | Register | FAQ | Members List | Calendar | Search | Today's Posts | Mark Forums Read |
|
#1
04-07-2008, 06:37 PM
| ||||
| ||||
 Hundreds of Yahoo! Spiders
Hello, Is it possible to limit the yahoo spiders crawling my web site? I'm getting 400-500 yahoo spiders (online almost all day long) per day and i can't understand why it's taking so long... My servers aren't overloaded and they are serving content really fast especially for US visitors.  Indexed pages for msxlabs.org: Google: 1,160,000 Yahoo: 493,916 Live: 3.140       |
|
#2
04-07-2008, 06:42 PM
| ||||
| ||||
|
Just found this page: How can I reduce the number of requests you make on my web site? - Yahoo! Search Help |
|
#3
04-07-2008, 07:06 PM
| ||||
| ||||
|
Updated the robots.txt file 10 minutes ago with the delay code and i have 150 yahoo bots instead of 410 (and getting smaller)  |
 |
| Tags |
| bots, google, live, spiders, yahoo |
| Thread Tools | |
| |
Similar Threads | ||||
| Thread | Thread Starter | Forum | Replies | Last Post |
| Spiders! | sohan | vBSEO Reviews | 5 | 11-11-2006 06:40 PM |
| Hundreds of yahoo slurp robots when vbseo is disabled | mcyates | General Discussion | 2 | 10-30-2006 01:26 PM |
| How Many Spiders...? | Paul Govier | General Discussion | 5 | 05-18-2006 04:44 PM |
| increasing yahoo slurp spiders | MentaL | General Discussion | 0 | 01-20-2006 12:08 AM |
| Yahoo only spiders external.php | caliman | General Discussion | 2 | 12-03-2005 03:42 PM |
All times are GMT -4. The time now is 12:48 AM.
