Results 1 to 6 of 6

Exclude Yahoo-Slurp by htaccess?

This is a discussion on Exclude Yahoo-Slurp by htaccess? within the Analysis: Traffic & SERPS forums, part of the vBulletin SEO Discussion category; Hello, I want to exclude the Yahoo-Bot from indexing pages via htaccess. What Condition should be used? PS: Yahoo Slurp ...

  1. #1
    Senior Member Silmarillion's Avatar
    Real Name
    Christian
    Join Date
    Jul 2005
    Location
    Germany
    Posts
    412
    Liked
    1 times

    Question Exclude Yahoo-Slurp by htaccess?

    Hello,

    I want to exclude the Yahoo-Bot from indexing pages via htaccess. What Condition should be used?

    PS: Yahoo Slurp causes a lot of Traffic and rising Serverloads. There are almost 50-100 Bots online 24h/day!
    The advantage isn`t noteworthy...Google and MSN are much better!

    Yours, Silmarillion

  2. #2
    Senior Member
    Real Name
    Keith Cohen
    Join Date
    Jul 2005
    Location
    Raleigh, NC USA
    Posts
    6,147
    Liked
    12 times
    Use robots.txt

  3. #3
    Senior Member Silmarillion's Avatar
    Real Name
    Christian
    Join Date
    Jul 2005
    Location
    Germany
    Posts
    412
    Liked
    1 times
    Robots.txt isn`t really able to prevent all Bots from visiting/indexing pages! What about Yahoo Slurp? Will it work?

    Yours, Silmarillion

  4. #4
    Senior Member
    Join Date
    Jul 2005
    Location
    Colorado
    Posts
    212
    Liked
    0 times
    AFAIK, slurp honors robots.txt, and it presents its user agent name upon spidering. If not (can anyone verify this?), you can use the vBulletin admin control panel to exclude the slurp IP block.
    Listings and promotion for hot rod and muscle car websites: The Crankshaft Coalition

  5. #5
    Senior Member
    Real Name
    Joseph Ward
    Join Date
    Jun 2005
    Posts
    23,847
    Liked
    32 times
    Blog Entries
    9
    @Silmarillion:

    Do you have a stats program that allows you to determine what amount of web visitors you are getting from Yahoo?

  6. #6
    Senior Member
    Real Name
    Keith Cohen
    Join Date
    Jul 2005
    Location
    Raleigh, NC USA
    Posts
    6,147
    Liked
    12 times
    Just as an example, I have a required profile dropdown field asking users where they found out about the site.

    Current stats:
    • Google: 37.9%
    • Friend: 13.5%
    • Other: 13.5%
    • Other Web Site: 13.3%
    • Yahoo!: 12.2%
    • MSN Search: 4.9%
    • Other Search Engine: 4.4%
    Last edited by Keith Cohen; 10-20-2005 at 12:25 PM.

Similar Threads

  1. yahoo slurp
    By xpfUK in forum General Discussion
    Replies: 7
    Last Post: 07-21-2007, 11:22 PM
  2. Yahoo doesnt like my site...
    By Barry in forum General Discussion
    Replies: 6
    Last Post: 02-04-2006, 03:24 AM
  3. increasing yahoo slurp spiders
    By MentaL in forum General Discussion
    Replies: 0
    Last Post: 01-20-2006, 01:08 AM
  4. Why does Yahoo hate me?
    By caliman in forum Off-Topic & Chit Chat
    Replies: 5
    Last Post: 11-18-2005, 03:31 AM
  5. VBSEO and YAHOO Slurp
    By MentaL in forum General Discussion
    Replies: 6
    Last Post: 11-02-2005, 11:11 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •