I have a group of forums at my site that are for professionals only, however I have it set up for visitors to be able to see the thread titles. I did that for 2 reasons: to show visitors that we have a very active professional contingent (so more pros will join and consumers will be assured) and to allow search engine spiders to index the content we have just in thread titles.
The problem is that spiders try to read the threads and get an error message and I'm not sure that's good for me. How can I prevent the spiders from trying to read threads in a private forum without barring access completely. I thought Google would be the easiest to control because I understood that they are the only spider that understands the wild-card character (*). Here is what my robots.txt looks like:
Someone has mentioned I might be able to do something with a meta tag, but I am clueless about that. What about a conditional? I can only guess that could lead to some heavy coding and I may not want to load my pages up with that. Any advice or guidance would be appreciated.Code:User-agent: GoogleBot Disallow: /website_map.php Disallow: /community/flooring-potpourri/* Disallow: /community/the-business-end-of-floorcovering-business/* Disallow: /community/carpet-installations/* Disallow: /community/sheet-vinyl-rubber-and-vct-installations/* Disallow: /community/hardwood-and-laminate-installations/* Disallow: /community/ceramic-and-stone-installations/* Disallow: /community/flooring-inspection-services/* Disallow: /community/cleaning-and-restoration-services/* Disallow: /community/tools-equipment-and-supplies/* Disallow: /community/product-announcements-news-and-reports/* Disallow: /community/general-construction-and-remodeling-workshop/* Disallow: /community/chiefcp/ Disallow: /community/announcement.php Disallow: /community/attachment.php Disallow: /community/calendar.php?do=add Disallow: /community/cron.php Disallow: /community/editpost.php Disallow: /community/joinrequests.php Disallow: /community/login.php Disallow: /community/memberlist.php? Disallow: /community/misc.php Disallow: /community/staffcp/ Disallow: /community/moderator.php Disallow: /community/newreply.php Disallow: /community/newthread.php Disallow: /community/online.php Disallow: /community/payments.php Disallow: /community/poll.php Disallow: /community/postings.php Disallow: /community/printthread.php Disallow: /community/private.php Disallow: /community/profile.php Disallow: /community/register.php Disallow: /community/report.php Disallow: /community/reputation.php Disallow: /community/sendtofriend.php Disallow: /community/search.php Disallow: /community/sendmessage.php Disallow: /community/showgroups.php Disallow: /community/showpost.php Disallow: /community/subscription.php Disallow: /community/subscriptions.php Disallow: /community/threadrate.php Disallow: /community/usercp.php User-agent: Yahoo! Slurp Disallow: /website_map.php Disallow: /community/chiefcp/ Disallow: /community/announcement.php Disallow: /community/attachment.php Disallow: /community/calendar.php?do=add Disallow: /community/cron.php Disallow: /community/editpost.php Disallow: /community/joinrequests.php Disallow: /community/login.php Disallow: /community/memberlist.php? Disallow: /community/misc.php Disallow: /community/staffcp/ Disallow: /community/moderator.php Disallow: /community/newreply.php Disallow: /community/newthread.php Disallow: /community/online.php Disallow: /community/payments.php Disallow: /community/poll.php Disallow: /community/postings.php Disallow: /community/printthread.php Disallow: /community/private.php Disallow: /community/profile.php Disallow: /community/register.php Disallow: /community/report.php Disallow: /community/reputation.php Disallow: /community/sendtofriend.php Disallow: /community/search.php Disallow: /community/sendmessage.php Disallow: /community/showgroups.php Disallow: /community/showpost.php Disallow: /community/subscription.php Disallow: /community/subscriptions.php Disallow: /community/threadrate.php Disallow: /community/usercp.php User-agent: * Disallow: /community/chiefcp/ Disallow: /community/announcement.php Disallow: /community/attachment.php Disallow: /community/calendar.php?do=add Disallow: /community/cron.php Disallow: /community/editpost.php Disallow: /community/joinrequests.php Disallow: /community/login.php Disallow: /community/memberlist.php? Disallow: /community/misc.php Disallow: /community/staffcp/ Disallow: /community/moderator.php Disallow: /community/newreply.php Disallow: /community/newthread.php Disallow: /community/online.php Disallow: /community/payments.php Disallow: /community/poll.php Disallow: /community/postings.php Disallow: /community/printthread.php Disallow: /community/private.php Disallow: /community/profile.php Disallow: /community/register.php Disallow: /community/report.php Disallow: /community/reputation.php Disallow: /community/sendtofriend.php Disallow: /community/search.php Disallow: /community/sendmessage.php Disallow: /community/showgroups.php Disallow: /community/showpost.php Disallow: /community/subscription.php Disallow: /community/subscriptions.php Disallow: /community/threadrate.php Disallow: /community/usercp.php
PS: can someone change my name to eJM? I have a feeling I will be becoming a multi-unit member and would prefer to go by my business name nic-name (business is enterpriseJM). Thanks in advance.


LinkBack URL
About LinkBacks





Reply With Quote