Results 1 to 2 of 2

Preventing spiders from browsing private threads

This is a discussion on Preventing spiders from browsing private threads within the Template Modifications forums, part of the vBulletin SEO Discussion category; I have a group of forums at my site that are for professionals only, however I have it set up ...

  1. #1
    eJM
    eJM is offline
    Senior Member eJM's Avatar
    Real Name
    Jim McClain
    Join Date
    May 2006
    Location
    teh Ether
    Posts
    311
    Liked
    2 times

    Question Preventing spiders from browsing private threads

    I have a group of forums at my site that are for professionals only, however I have it set up for visitors to be able to see the thread titles. I did that for 2 reasons: to show visitors that we have a very active professional contingent (so more pros will join and consumers will be assured) and to allow search engine spiders to index the content we have just in thread titles.

    The problem is that spiders try to read the threads and get an error message and I'm not sure that's good for me. How can I prevent the spiders from trying to read threads in a private forum without barring access completely. I thought Google would be the easiest to control because I understood that they are the only spider that understands the wild-card character (*). Here is what my robots.txt looks like:
    Code:
    User-agent: GoogleBot
    Disallow: /website_map.php
    Disallow: /community/flooring-potpourri/*
    Disallow: /community/the-business-end-of-floorcovering-business/*
    Disallow: /community/carpet-installations/*
    Disallow: /community/sheet-vinyl-rubber-and-vct-installations/*
    Disallow: /community/hardwood-and-laminate-installations/*
    Disallow: /community/ceramic-and-stone-installations/*
    Disallow: /community/flooring-inspection-services/*
    Disallow: /community/cleaning-and-restoration-services/*
    Disallow: /community/tools-equipment-and-supplies/*
    Disallow: /community/product-announcements-news-and-reports/*
    Disallow: /community/general-construction-and-remodeling-workshop/*
    Disallow: /community/chiefcp/
    Disallow: /community/announcement.php
    Disallow: /community/attachment.php
    Disallow: /community/calendar.php?do=add
    Disallow: /community/cron.php
    Disallow: /community/editpost.php
    Disallow: /community/joinrequests.php
    Disallow: /community/login.php
    Disallow: /community/memberlist.php?
    Disallow: /community/misc.php
    Disallow: /community/staffcp/
    Disallow: /community/moderator.php
    Disallow: /community/newreply.php
    Disallow: /community/newthread.php
    Disallow: /community/online.php
    Disallow: /community/payments.php
    Disallow: /community/poll.php
    Disallow: /community/postings.php
    Disallow: /community/printthread.php
    Disallow: /community/private.php
    Disallow: /community/profile.php
    Disallow: /community/register.php
    Disallow: /community/report.php
    Disallow: /community/reputation.php
    Disallow: /community/sendtofriend.php
    Disallow: /community/search.php
    Disallow: /community/sendmessage.php
    Disallow: /community/showgroups.php
    Disallow: /community/showpost.php
    Disallow: /community/subscription.php
    Disallow: /community/subscriptions.php
    Disallow: /community/threadrate.php
    Disallow: /community/usercp.php
    
    User-agent: Yahoo! Slurp
    Disallow: /website_map.php
    Disallow: /community/chiefcp/
    Disallow: /community/announcement.php
    Disallow: /community/attachment.php
    Disallow: /community/calendar.php?do=add
    Disallow: /community/cron.php
    Disallow: /community/editpost.php
    Disallow: /community/joinrequests.php
    Disallow: /community/login.php
    Disallow: /community/memberlist.php?
    Disallow: /community/misc.php
    Disallow: /community/staffcp/
    Disallow: /community/moderator.php
    Disallow: /community/newreply.php
    Disallow: /community/newthread.php
    Disallow: /community/online.php
    Disallow: /community/payments.php
    Disallow: /community/poll.php
    Disallow: /community/postings.php
    Disallow: /community/printthread.php
    Disallow: /community/private.php
    Disallow: /community/profile.php
    Disallow: /community/register.php
    Disallow: /community/report.php
    Disallow: /community/reputation.php
    Disallow: /community/sendtofriend.php
    Disallow: /community/search.php
    Disallow: /community/sendmessage.php
    Disallow: /community/showgroups.php
    Disallow: /community/showpost.php
    Disallow: /community/subscription.php
    Disallow: /community/subscriptions.php
    Disallow: /community/threadrate.php
    Disallow: /community/usercp.php
    
    User-agent: *
    Disallow: /community/chiefcp/
    Disallow: /community/announcement.php
    Disallow: /community/attachment.php
    Disallow: /community/calendar.php?do=add
    Disallow: /community/cron.php
    Disallow: /community/editpost.php
    Disallow: /community/joinrequests.php
    Disallow: /community/login.php
    Disallow: /community/memberlist.php?
    Disallow: /community/misc.php
    Disallow: /community/staffcp/
    Disallow: /community/moderator.php
    Disallow: /community/newreply.php
    Disallow: /community/newthread.php
    Disallow: /community/online.php
    Disallow: /community/payments.php
    Disallow: /community/poll.php
    Disallow: /community/postings.php
    Disallow: /community/printthread.php
    Disallow: /community/private.php
    Disallow: /community/profile.php
    Disallow: /community/register.php
    Disallow: /community/report.php
    Disallow: /community/reputation.php
    Disallow: /community/sendtofriend.php
    Disallow: /community/search.php
    Disallow: /community/sendmessage.php
    Disallow: /community/showgroups.php
    Disallow: /community/showpost.php
    Disallow: /community/subscription.php
    Disallow: /community/subscriptions.php
    Disallow: /community/threadrate.php
    Disallow: /community/usercp.php
    Someone has mentioned I might be able to do something with a meta tag, but I am clueless about that. What about a conditional? I can only guess that could lead to some heavy coding and I may not want to load my pages up with that. Any advice or guidance would be appreciated.

    PS: can someone change my name to eJM? I have a feeling I will be becoming a multi-unit member and would prefer to go by my business name nic-name (business is enterpriseJM). Thanks in advance.

  2. #2
    Junior Member
    Real Name
    eXtreme
    Join Date
    Nov 2006
    Posts
    16
    Liked
    0 times
    i heard they use to

    no follow tag or something

Similar Threads

  1. i think something is wrong....
    By briansol in forum General Discussion
    Replies: 3
    Last Post: 08-20-2006, 06:10 PM
  2. Replies: 8
    Last Post: 08-15-2006, 04:10 AM
  3. Replies: 3
    Last Post: 08-02-2006, 07:37 AM
  4. whoa, tons of errors
    By mindhunter77 in forum Troubleshooting
    Replies: 5
    Last Post: 03-10-2006, 03:39 AM
  5. Compression Error
    By Lazer in forum Troubleshooting
    Replies: 19
    Last Post: 03-03-2006, 01:23 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •