Results 1 to 3 of 3

Robots file review and prevent forum viewing

This is a discussion on Robots file review and prevent forum viewing within the General Discussion forums, part of the vBSEO SEO Plugin category; I didn't have a robots.txt file before, but now I think I should have one, right? Here is what I'm ...

  1. #1
    Junior Member
    Real Name
    Jon
    Join Date
    Apr 2009
    Posts
    25
    Liked
    0 times

    Robots file review and prevent forum viewing

    I didn't have a robots.txt file before, but now I think I should have one, right? Here is what I'm gonna put in it. I've implemented VBSEO sitemap generator 3.0 as well. My question is, how do I prevent a bot from attempting to access one of my forums? Right now I see the red stop sign and they are being blocked, but can you make it so that the bots wont be able to see a specific forum thus not attempting to access it? Thanks in advance

    Code:
    User-agent: *
    Disallow: /admincp/
    Disallow: /cgi-bin/
    Disallow: /clientscript/
    Disallow: /includes/
    Disallow: /install/
    Disallow: /modcp/
    Disallow: /subscription.php
    Disallow: /payments.php
    Disallow: /faq.php
    Disallow: /calendar.php
    Disallow: /search.php
    Disallow: /private.php
    Disallow: /online.php
    Disallow: /sendmessage.php
    Disallow: /sendmessage.php?do=
    Disallow: /showgroups.php
    Disallow: /reputation.php
    Disallow: /report.php
    Disallow: /threadrate.php
    Disallow: /postings.php
    Disallow: /newthread.php
    Disallow: /newreply.php
    Disallow: /register.php
    Disallow: /login.php
    Disallow: /faq.php
    Disallow: /image.php
    Disallow: /cron.php
    Disallow: /joinrequests.php
    Disallow: /printthread.php
    Disallow: /showpost.php
    Disallow: /archive/
    Disallow: /sitemap/

    Also on my forum, the whos online list, the google bots never are in active forums. They are consistently trying to access forums that have no permission on them.

  2. #2
    vBSEO Staff Brian Cummiskey's Avatar
    Real Name
    Brian Cummiskey
    Join Date
    Jul 2009
    Location
    btwn NYC and Boston
    Posts
    12,789
    Liked
    657 times
    Blog Entries
    2

  3. #3
    vBSEO Staff Andrés Durán Hewitt's Avatar
    Real Name
    Andrés Durán
    Join Date
    Jul 2009
    Location
    Costa Rica
    Posts
    3,393
    Liked
    411 times
    Blog Entries
    1
    Hi Jon,

    Besides the information on the link posted above, you should not add your "admincp/", "modcp/" nor "install/" directories for security purposes. Please see for more info: Easy Security Tips for vBSEO customers
    Andrés Durán / Crawlability Inc.
    ˇvBSEO 3.6.0 GOLD Liberado!
    Inaugurando el NUEVO vBSEO Sitemap Generator 3.0. - ˇAHORA disponible para Clientes de vBSEO!

    Síguenos en: Facebook | Síguenos en: Twitter


Similar Threads

  1. vBulletin 4.x Is this an accurate robots file?
    By rubyguava in forum Troubleshooting
    Replies: 6
    Last Post: 06-21-2011, 12:32 AM
  2. Testing robots.txt file...
    By Sonnie in forum General Discussion
    Replies: 8
    Last Post: 01-06-2011, 10:59 PM
  3. Is my robots.tx file ok?
    By swiftor in forum General Discussion
    Replies: 4
    Last Post: 10-11-2010, 05:15 AM
  4. Robots.txt file changes
    By Blind Dragon in forum General Discussion
    Replies: 3
    Last Post: 09-03-2009, 01:00 PM
  5. Please review my robots.txt file...
    By Sonnie in forum General Discussion
    Replies: 11
    Last Post: 09-08-2006, 03:30 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •