I didn't have a robots.txt file before, but now I think I should have one, right? Here is what I'm gonna put in it. I've implemented VBSEO sitemap generator 3.0 as well. My question is, how do I prevent a bot from attempting to access one of my forums? Right now I see the red stop sign and they are being blocked, but can you make it so that the bots wont be able to see a specific forum thus not attempting to access it? Thanks in advance
Code:User-agent: * Disallow: /admincp/ Disallow: /cgi-bin/ Disallow: /clientscript/ Disallow: /includes/ Disallow: /install/ Disallow: /modcp/ Disallow: /subscription.php Disallow: /payments.php Disallow: /faq.php Disallow: /calendar.php Disallow: /search.php Disallow: /private.php Disallow: /online.php Disallow: /sendmessage.php Disallow: /sendmessage.php?do= Disallow: /showgroups.php Disallow: /reputation.php Disallow: /report.php Disallow: /threadrate.php Disallow: /postings.php Disallow: /newthread.php Disallow: /newreply.php Disallow: /register.php Disallow: /login.php Disallow: /faq.php Disallow: /image.php Disallow: /cron.php Disallow: /joinrequests.php Disallow: /printthread.php Disallow: /showpost.php Disallow: /archive/ Disallow: /sitemap/
Also on my forum, the whos online list, the google bots never are in active forums. They are consistently trying to access forums that have no permission on them.


LinkBack URL
About LinkBacks





Reply With Quote
