vBulletin 4, the most powerful community software + vBSEO 3.5, the ultimate SEO solution = Your ultimate platform for 2010 and beyond. Click below to learn more.

Results 1 to 2 of 2

My robots.txt - what is in there / what isn't there?

This is a discussion on My robots.txt - what is in there / what isn't there? within the General Discussion forums, part of the vBSEO SEO Plugin category; This is my robots.txt User-agent: * Disallow: /images/ Disallow: /faq.php Disallow: /attachment.php Disallow: /avatar.php Disallow: /cron.php Disallow: /editpost.php Disallow: /calendar.php ...

  1. #1
    Senior Member
    Real Name
    Admin
    Join Date
    Nov 2006
    Posts
    172

    My robots.txt - what is in there / what isn't there?

    This is my robots.txt
    User-agent: *
    Disallow: /images/
    Disallow: /faq.php
    Disallow: /attachment.php
    Disallow: /avatar.php
    Disallow: /cron.php
    Disallow: /editpost.php
    Disallow: /calendar.php
    Disallow: /member.php
    Disallow: /memberlist.php
    Disallow: /printthread.php
    Disallow: /sendmessage.php
    Disallow: /register.php
    Disallow: /sendtofriend.php
    Disallow: /login.php
    Disallow: /newreply.php
    Disallow: /newthread.php
    Disallow: /online.php
    Disallow: /search.php
    Disallow: /subscription.php
    Disallow: /threadrate.php
    Disallow: /private.php
    Disallow: /misc.php
    Disallow: /poll.php
    Disallow: /showpost.php
    Disallow: /profile.php
    Disallow: /usercp.php
    Disallow: /showgroups.php
    Disallow: /cmps_index.php
    1. question:
    Is all of the above correct?

    2. question:
    As I read somewhere here, the following should also be in the robots.txt:

    ajax.php
    global.php
    image.php
    inlinemod.php
    joinrequests.php
    moderator.php
    newattachment.php
    postings.php
    report.php
    reputation.php
    usernote.php
    Is that correct?

    Thanks!

    Marcel
    Last edited by symptome; 08-13-2008 at 03:21 PM.

  2. #2
    Senior Member briansol's Avatar
    Real Name
    Brian
    Join Date
    Apr 2006
    Location
    Central CT, USA
    Posts
    6,983
    never EVER list secure/private directories in your robots file



    remove those, and rename them as well for best security.


    what you disallow greatly depends on what you chose to re-write and what is left as .php files.

    me, i re-write everything i want indexed, and disallow *.php, *.aspx, *.jsp and all the rest to confuse
    Last edited by briansol; 08-13-2008 at 03:27 PM.

Similar Threads

  1. robots.txt
    By Zenith in forum General Discussion
    Replies: 60
    Last Post: 11-04-2009, 03:44 PM
  2. Robots.txt help
    By Dave Hybrid in forum General Discussion
    Replies: 2
    Last Post: 05-29-2007, 10:16 AM
  3. The best robots.txt
    By MaestroX in forum General Discussion
    Replies: 1
    Last Post: 01-28-2007, 04:53 PM
  4. No Robots.txt
    By friendly in forum Troubleshooting
    Replies: 1
    Last Post: 10-26-2006, 10:30 AM
  5. How is my robots.txt?
    By libertylounge in forum Troubleshooting
    Replies: 6
    Last Post: 08-26-2006, 12:17 PM