Results 1 to 5 of 5

Robots.txt

This is a discussion on Robots.txt within the General Discussion forums, part of the vBulletin SEO Discussion category; I'm looking for thoughts/opinions on what should be in robots.txt for my forum. At the moment, I have: User-agent: * ...

  1. #1
    Junior Member
    Real Name
    Eric Sizemore
    Join Date
    Aug 2006
    Location
    Kentucky
    Posts
    6
    Liked
    0 times

    Robots.txt

    I'm looking for thoughts/opinions on what should be in robots.txt for my forum. At the moment, I have:

    User-agent: *
    Disallow: /acp/
    Disallow: /archive/
    Disallow: /cgi-bin/
    Disallow: /clientscript/
    Disallow: /cpsytles/
    Disallow: /customavatars/
    Disallow: /customprofilepics/
    Disallow: /images/
    Disallow: /includes/
    Disallow: /install/
    Disallow: /mcp/
    Disallow: /signaturepics/
    Disallow: /vbseo_sitemap/
    Disallow: ajax.php
    Disallow: album.php
    Disallow: announcement.php
    Disallow: attachment.php
    Disallow: calendar.php
    Disallow: clear.gif
    Disallow: converse.php
    Disallow: cron.php
    Disallow: editpost.php
    Disallow: faq.php
    Disallow: forumdisplay.php
    Disallow: global.php
    Disallow: group.php
    Disallow: group_inlinemod.php
    Disallow: image.php
    Disallow: infraction.php
    Disallow: inlinemod.php
    Disallow: joinrequests.php
    Disallow: login.php
    Disallow: member.php
    Disallow: member_inlinemod.php
    Disallow: memberlist.php
    Disallow: misc.php
    Disallow: moderation.php
    Disallow: moderator.php
    Disallow: newattachment.php
    Disallow: newreply.php
    Disallow: newthread.php
    Disallow: online.php
    Disallow: payment_gateway.php
    Disallow: payments.php
    Disallow: picture.php
    Disallow: picture_inlinemod.php
    Disallow: picturecomment.php
    Disallow: poll.php
    Disallow: posthistory.php
    Disallow: postings.php
    Disallow: printthread.php
    Disallow: private.php
    Disallow: profile.php
    Disallow: register.php
    Disallow: report.php
    Disallow: reputation.php
    Disallow: search.php
    Disallow: sendmessage.php
    Disallow: showgroups.php
    Disallow: showpost.php
    Disallow: showthread.php
    Disallow: subscription.php
    Disallow: tags.php
    Disallow: threadrate.php
    Disallow: threadtag.php
    Disallow: usercp.php
    Disallow: usernote.php
    Disallow: vbseo.php
    Disallow: vbseocpform.php
    Disallow: visitormessage.php

  2. #2
    Senior Member briansol's Avatar
    Real Name
    Brian
    Join Date
    Apr 2006
    Location
    Central CT, USA
    Posts
    6,981
    Liked
    8 times
    for security reasons, NEVER list your modcp and admincp directories. Changing them is a moot point if you list them in the robots file.

    I used to have everything listed out.
    now, since everything i use that i want indexed is re-written, i simply blocked *.php

  3. #3
    Junior Member
    Real Name
    Eric Sizemore
    Join Date
    Aug 2006
    Location
    Kentucky
    Posts
    6
    Liked
    0 times
    Thanks brian I had actually not realized until you mentioned it that I included my mod and admin directories, oops. I'll remove those. I have everything rewritten now, so it would probably best to:

    Disallow: *.php


  4. #4
    Senior Member briansol's Avatar
    Real Name
    Brian
    Join Date
    Apr 2006
    Location
    Central CT, USA
    Posts
    6,981
    Liked
    8 times
    even better,
    fool would-be hackers.

    block evenrtyhing you don't have....


    Disallow: *.asp
    Disallow: *.php
    Disallow: *.aspx
    Disallow: *.jsp
    Disallow: *.cfm
    Disallow: *.cgi
    Disallow: *.pl



  5. #5
    Senior Member Bonus Paradise's Avatar
    Real Name
    Marina
    Join Date
    Jun 2008
    Location
    Europe
    Posts
    181
    Liked
    0 times
    Quote Originally Posted by briansol View Post
    even better,
    fool would-be hackers.

    block evenrtyhing you don't have....


    Disallow: *.asp
    Disallow: *.php
    Disallow: *.aspx
    Disallow: *.jsp
    Disallow: *.cfm
    Disallow: *.cgi
    Disallow: *.pl


    Hi there briansol
    I am interested in this
    What do you think can i just add this so how you posted it in my robots txt?
    my forum is Bonus Paradise Gambling Forum
    of course i like googe still indexing all what is important
    would appreciate if you or anyone else could help me with the robots txt
    I am a bit worried to do something wrong

    Regards
    Marina

Similar Threads

  1. robots.txt
    By Zenith in forum General Discussion
    Replies: 64
    Last Post: 12-01-2010, 07:52 PM
  2. About robots.txt
    By Diver in forum General Discussion
    Replies: 1
    Last Post: 02-18-2008, 07:18 PM
  3. robots.txt
    By wzrdsucks in forum General Discussion
    Replies: 4
    Last Post: 11-21-2007, 03:42 PM
  4. How is my robots.txt?
    By libertylounge in forum Troubleshooting
    Replies: 6
    Last Post: 08-26-2006, 12:17 PM

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •