Results 1 to 2 of 2

My robots.txt - what is in there / what isn't there?

This is a discussion on My robots.txt - what is in there / what isn't there? within the General Discussion forums, part of the vBSEO SEO Plugin category; This is my robots.txt User-agent: * Disallow: /images/ Disallow: /faq.php Disallow: /attachment.php Disallow: /avatar.php Disallow: /cron.php Disallow: /editpost.php Disallow: /calendar.php ...

  1. #1
    Senior Member
    Real Name
    Admin
    Join Date
    Nov 2006
    Posts
    247
    Liked
    0 times

    My robots.txt - what is in there / what isn't there?

    This is my robots.txt
    User-agent: *
    Disallow: /images/
    Disallow: /faq.php
    Disallow: /attachment.php
    Disallow: /avatar.php
    Disallow: /cron.php
    Disallow: /editpost.php
    Disallow: /calendar.php
    Disallow: /member.php
    Disallow: /memberlist.php
    Disallow: /printthread.php
    Disallow: /sendmessage.php
    Disallow: /register.php
    Disallow: /sendtofriend.php
    Disallow: /login.php
    Disallow: /newreply.php
    Disallow: /newthread.php
    Disallow: /online.php
    Disallow: /search.php
    Disallow: /subscription.php
    Disallow: /threadrate.php
    Disallow: /private.php
    Disallow: /misc.php
    Disallow: /poll.php
    Disallow: /showpost.php
    Disallow: /profile.php
    Disallow: /usercp.php
    Disallow: /showgroups.php
    Disallow: /cmps_index.php
    1. question:
    Is all of the above correct?

    2. question:
    As I read somewhere here, the following should also be in the robots.txt:

    ajax.php
    global.php
    image.php
    inlinemod.php
    joinrequests.php
    moderator.php
    newattachment.php
    postings.php
    report.php
    reputation.php
    usernote.php
    Is that correct?

    Thanks!

    Marcel
    Last edited by symptome; 08-13-2008 at 04:21 PM.

  2. #2
    Senior Member briansol's Avatar
    Real Name
    Brian
    Join Date
    Apr 2006
    Location
    Central CT, USA
    Posts
    6,981
    Liked
    8 times
    never EVER list secure/private directories in your robots file



    remove those, and rename them as well for best security.


    what you disallow greatly depends on what you chose to re-write and what is left as .php files.

    me, i re-write everything i want indexed, and disallow *.php, *.aspx, *.jsp and all the rest to confuse
    Last edited by briansol; 08-13-2008 at 04:27 PM.

Similar Threads

  1. robots.txt
    By Zenith in forum General Discussion
    Replies: 64
    Last Post: 12-01-2010, 08:52 PM
  2. Robots.txt help
    By Dave Hybrid in forum General Discussion
    Replies: 2
    Last Post: 05-29-2007, 11:16 AM
  3. The best robots.txt
    By MaestroX in forum General Discussion
    Replies: 1
    Last Post: 01-28-2007, 05:53 PM
  4. No Robots.txt
    By friendly in forum Troubleshooting
    Replies: 1
    Last Post: 10-26-2006, 11:30 AM
  5. How is my robots.txt?
    By libertylounge in forum Troubleshooting
    Replies: 6
    Last Post: 08-26-2006, 01:17 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •