Results 1 to 4 of 4

Best way to restrict spiders from certain forums?

This is a discussion on Best way to restrict spiders from certain forums? within the General Discussion forums, part of the vBulletin SEO Discussion category; I have a few forums that I want to restrict the google/yahoo/etc. spiders from crawling. What is the best way ...

  1. #1
    Junior Member
    Real Name
    Tim
    Join Date
    Jul 2008
    Posts
    18
    Liked
    0 times

    Best way to restrict spiders from certain forums?

    I have a few forums that I want to restrict the google/yahoo/etc. spiders from crawling. What is the best way to do this? I have already removed them from the sitemap generation, but google still wants to index them (and I'm trying to funnel google toward other content). Should I just add the forums URLS to robots.txt, or is there a better way?

  2. #2
    vBSEO Staff Brian Cummiskey's Avatar
    Real Name
    Brian Cummiskey
    Join Date
    Jul 2009
    Location
    btwn NYC and Boston
    Posts
    12,789
    Liked
    657 times
    Blog Entries
    2
    You can either use robots.txt or set the fourm permissions so that guests can't view them, only members.

  3. #3
    Junior Member
    Real Name
    Tim
    Join Date
    Jul 2008
    Posts
    18
    Liked
    0 times
    I want guests to be able to see them...I just don't want google wasting time there. Google is obsessed with some forums on my site that add almost zero SEO value. So it sounds like robots.txt is the only way to go? Should I use the VBSEO URLs in the robots.txt or the default vBulletin urls?

  4. #4
    vBSEO Staff Brian Cummiskey's Avatar
    Real Name
    Brian Cummiskey
    Join Date
    Jul 2009
    Location
    btwn NYC and Boston
    Posts
    12,789
    Liked
    657 times
    Blog Entries
    2
    Yes, in this case, robots.txt is the only safe way to block it.

    You should use
    disallow: /nice-forum/
    style

Similar Threads

  1. How to restrict guests to one page only on threads
    By Cromulent in forum Template Modifications
    Replies: 2
    Last Post: 10-19-2007, 09:22 PM
  2. Spiders!
    By sohan in forum Analysis: Traffic & SERPS
    Replies: 5
    Last Post: 11-11-2006, 06:40 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •