Results 1 to 8 of 8

Help with my robots.txt

This is a discussion on Help with my robots.txt within the Troubleshooting forums, part of the vBSEO SEO Plugin category; Hi, I know a lot of people simply "Disallow: *.php" in their robots.txt file which is a lot easier than ...

  1. #1
    Member
    Real Name
    JPetrillo
    Join Date
    Jun 2009
    Location
    CT
    Posts
    65
    Liked
    0 times

    Help with my robots.txt

    Hi,

    I know a lot of people simply "Disallow: *.php" in their robots.txt file which is a lot easier than making a huge list of all the individual files, but I'm unable to do it the easy way since I have *.php files that I do not want blocked, like this one for example.

    So I just put together my robots.txt, but before I upload it I want to make sure it's alright:

    User-Agent: *
    Disallow: /forum/archive/
    Disallow: /forum/clientscript/
    Disallow: /forum/cpstyles/
    Disallow: /forum/customavatars/
    Disallow: /forum/customgroupicons/
    Disallow: /forum/customprofilepics/
    Disallow: /forum/images/
    Disallow: /forum/includes/
    Disallow: /forum/install/
    Disallow: /forum/modules/
    Disallow: /forum/signaturepics/
    Disallow: /forum/tmp/
    Disallow: /forum/vbseo_sitemap/
    Disallow: /forum/ajax.php
    Disallow: /forum/album.php
    Disallow: /forum/announcement.php
    Disallow: /forum/attachment.php
    Disallow: /forum/calendar.php
    Disallow: /forum/clear.gif
    Disallow: /forum/converse.php
    Disallow: /forum/cron.php
    Disallow: /forum/editpost.php
    Disallow: /forum/external.php
    Disallow: /forum/faq.php
    Disallow: /forum/favicon.ico
    Disallow: /forum/forumdisplay.php
    Disallow: /forum/global.php
    Disallow: /forum/group.php
    Disallow: /forum/group_inlinemod.php
    Disallow: /forum/groupsubscription.php
    Disallow: /forum/gxboxlive.php
    Disallow: /forum/image.php
    Disallow: /forum/index.php
    Disallow: /forum/infraction.php
    Disallow: /forum/inlinemod.php
    Disallow: /forum/joinrequests.php
    Disallow: /forum/LICENSE
    Disallow: /forum/login.php
    Disallow: /forum/member.php
    Disallow: /forum/member_inlinemod.php
    Disallow: /forum/memberlist.php
    Disallow: /forum/misc.php
    Disallow: /forum/moderation.php
    Disallow: /forum/moderator.php
    Disallow: /forum/newattachment.php
    Disallow: /forum/newreply.php
    Disallow: /forum/newthread.php
    Disallow: /forum/online.php
    Disallow: /forum/payment_gateway.php
    Disallow: /forum/payments.php
    Disallow: /forum/picture.php
    Disallow: /forum/picture_inlinemod.php
    Disallow: /forum/picturecomment.php
    Disallow: /forum/poll.php
    Disallow: /forum/posthistory.php
    Disallow: /forum/postings.php
    Disallow: /forum/printthread.php
    Disallow: /forum/privacy.php
    Disallow: /forum/private.php
    Disallow: /forum/profile.php
    Disallow: /forum/register.php
    Disallow: /forum/report.php
    Disallow: /forum/reputation.php
    Disallow: /forum/search.php
    Disallow: /forum/sendmessage.php
    Disallow: /forum/showgroups.php
    Disallow: /forum/showpost.php
    Disallow: /forum/showthread.php
    Disallow: /forum/subscription.php
    Disallow: /forum/tags.php
    Disallow: /forum/threadrate.php
    Disallow: /forum/threadtag.php
    Disallow: /forum/usercp.php
    Disallow: /forum/usernote.php
    Disallow: /forum/vbseo.php
    Disallow: /forum/vbseocp.php
    Disallow: /forum/vbseocpform.php
    Disallow: /forum/visitormessage.php
    Disallow: /forum/xperience.php

    Sitemap: http://www.gamersthrone.com/forum/sitemap_index.xml.g
    The lines in red are the ones I'm really not sure about, especially the Privacy and gxboxlive files.

    Can someone please confirm before I upload this?

    Thank you!

  2. #2
    Senior Member Brandon Sheley's Avatar
    Real Name
    Brandon Sheley
    Join Date
    Oct 2005
    Location
    Kansas
    Posts
    2,348
    Liked
    20 times
    Blog Entries
    1
    I always just delete the LICENSE file
    and I wouldn't disallow bots from my index.php file
    and I wouldn't worry about the favoricon
    other than that, it looks "okay"
    here is mine if you want to compare
    http://forum.vbulletinsetup.com/robots.txt
    My forums: General Forums | Admin Talk (running xenforo)

  3. #3
    Member
    Real Name
    JPetrillo
    Join Date
    Jun 2009
    Location
    CT
    Posts
    65
    Liked
    0 times
    Okay, thanks for the reply. I finally uploaded it

    I removed the following:

    Disallow: /forum/index.php
    Disallow: /forum/privacy.php
    Disallow: /forum/gxboxlive.php

    ...this way there wont be any issues viewing the forum, the privacy statement, or my Xbox Live Leaderboard.

    I added the following:

    Disallow: /forum/tags

    Now it looks like this...

    User-Agent: *
    Disallow: /forum/archive/
    Disallow: /forum/clientscript/
    Disallow: /forum/cpstyles/
    Disallow: /forum/customavatars/
    Disallow: /forum/customgroupicons/
    Disallow: /forum/customprofilepics/
    Disallow: /forum/images/
    Disallow: /forum/includes/
    Disallow: /forum/install/
    Disallow: /forum/modules/
    Disallow: /forum/signaturepics/
    Disallow: /forum/tmp/
    Disallow: /forum/vbseo_sitemap/
    Disallow: /forum/tags
    Disallow: /forum/ajax.php
    Disallow: /forum/album.php
    Disallow: /forum/announcement.php
    Disallow: /forum/attachment.php
    Disallow: /forum/calendar.php
    Disallow: /forum/clear.gif
    Disallow: /forum/converse.php
    Disallow: /forum/cron.php
    Disallow: /forum/editpost.php
    Disallow: /forum/external.php
    Disallow: /forum/faq.php
    Disallow: /forum/favicon.ico
    Disallow: /forum/forumdisplay.php
    Disallow: /forum/global.php
    Disallow: /forum/group.php
    Disallow: /forum/group_inlinemod.php
    Disallow: /forum/groupsubscription.php
    Disallow: /forum/image.php
    Disallow: /forum/infraction.php
    Disallow: /forum/inlinemod.php
    Disallow: /forum/joinrequests.php
    Disallow: /forum/login.php
    Disallow: /forum/member.php
    Disallow: /forum/member_inlinemod.php
    Disallow: /forum/memberlist.php
    Disallow: /forum/misc.php
    Disallow: /forum/moderation.php
    Disallow: /forum/moderator.php
    Disallow: /forum/newattachment.php
    Disallow: /forum/newreply.php
    Disallow: /forum/newthread.php
    Disallow: /forum/online.php
    Disallow: /forum/payment_gateway.php
    Disallow: /forum/payments.php
    Disallow: /forum/picture.php
    Disallow: /forum/picture_inlinemod.php
    Disallow: /forum/picturecomment.php
    Disallow: /forum/poll.php
    Disallow: /forum/posthistory.php
    Disallow: /forum/postings.php
    Disallow: /forum/printthread.php
    Disallow: /forum/private.php
    Disallow: /forum/profile.php
    Disallow: /forum/register.php
    Disallow: /forum/report.php
    Disallow: /forum/reputation.php
    Disallow: /forum/search.php
    Disallow: /forum/sendmessage.php
    Disallow: /forum/showgroups.php
    Disallow: /forum/showpost.php
    Disallow: /forum/showthread.php
    Disallow: /forum/subscription.php
    Disallow: /forum/tags.php
    Disallow: /forum/threadrate.php
    Disallow: /forum/threadtag.php
    Disallow: /forum/usercp.php
    Disallow: /forum/usernote.php
    Disallow: /forum/vbseo.php
    Disallow: /forum/vbseocp.php
    Disallow: /forum/vbseocpform.php
    Disallow: /forum/visitormessage.php
    Disallow: /forum/xperience.php

    Sitemap: http://www.gamersthrone.com/forum/sitemap_index.xml.gz

  4. #4
    Senior Member
    Real Name
    Martyn Day
    Join Date
    Dec 2005
    Location
    Kent - UK
    Posts
    650
    Liked
    0 times
    Blog Entries
    1
    why not just disallow .php? would save alot of issues..... rather then publicly show the files you wish not be indexed, as some crawling bots may ignore the robot.txt

  5. #5
    Member
    Real Name
    JPetrillo
    Join Date
    Jun 2009
    Location
    CT
    Posts
    65
    Liked
    0 times
    Quote Originally Posted by Martyn View Post
    why not just disallow .php? would save alot of issues..... rather then publicly show the files you wish not be indexed, as some crawling bots may ignore the robot.txt
    Because...

    Quote Originally Posted by JPetrillo View Post
    Hi,

    I know a lot of people simply "Disallow: *.php" in their robots.txt file which is a lot easier than making a huge list of all the individual files, but I'm unable to do it the easy way since I have *.php files that I do not want blocked, like this one for example.

    ...

  6. #6
    Senior Member
    Real Name
    Martyn Day
    Join Date
    Dec 2005
    Location
    Kent - UK
    Posts
    650
    Liked
    0 times
    Blog Entries
    1
    why not setit up within vbseo so your reviews will be either end with a slash or .html ??

    you could make your link
    Code:
    http://www.gamersthrone.com/forum/reviews/xbox360/ghostbusters/ 
    or end with .html
    http://www.gamersthrone.com/forum/reviews/xbox360/ghostbusters.html
    or even!
    http://www.gamersthrone.com/forum/xbox360review-ghostbusters.html
    rather then the http://www.gamersthrone.com/forum/xb...ostbusters.php

    also you can make it easier on your self by doing it within the template and just calling misc.php rather then creating another .php file and template to every review you do, this would be more sufficient, and think down the line when you have 100 reviews, thats 100 .php files your have in your root folder!

  7. #7
    Senior Member
    Real Name
    Stavros
    Join Date
    Sep 2007
    Posts
    148
    Liked
    0 times
    Quote Originally Posted by JPetrillo View Post
    I removed the following:

    Disallow: /forum/privacy.php
    Is there any particular reason that you want privacy.php to be indexed?

  8. #8
    Member
    Real Name
    JPetrillo
    Join Date
    Jun 2009
    Location
    CT
    Posts
    65
    Liked
    0 times
    Quote Originally Posted by Martyn View Post
    why not setit up within vbseo so your reviews will be either end with a slash or .html ??

    you could make your link
    Code:
    http://www.gamersthrone.com/forum/reviews/xbox360/ghostbusters/ 
    or end with .html
    http://www.gamersthrone.com/forum/reviews/xbox360/ghostbusters.html
    or even!
    http://www.gamersthrone.com/forum/xbox360review-ghostbusters.html
    rather then the GamersThrone: Video Game Forums

    also you can make it easier on your self by doing it within the template and just calling misc.php rather then creating another .php file and template to every review you do, this would be more sufficient, and think down the line when you have 100 reviews, thats 100 .php files your have in your root folder!
    Wow that's a great idea! How do I setup the URL's like that? I have no idea how to do that...

    Quote Originally Posted by Zagis View Post
    Is there any particular reason that you want privacy.php to be indexed?
    I don't know... I thought it was important? I should block that too?

Similar Threads

  1. robots.txt
    By Bram in forum URL Rewrite Settings
    Replies: 7
    Last Post: 06-23-2009, 11:48 AM
  2. Redirecting /forums/robots.txt to /robots.txt - Is it good?
    By MadK in forum Custom Rewrite Rules
    Replies: 6
    Last Post: 08-22-2008, 06:29 PM
  3. My robots.txt - what is in there / what isn't there?
    By symptome in forum General Discussion
    Replies: 1
    Last Post: 08-13-2008, 03:15 PM
  4. robots.txt
    By frank678 in forum General Discussion
    Replies: 2
    Last Post: 03-03-2008, 02:12 AM
  5. Robots.txt Help
    By soletrader in forum General Discussion
    Replies: 7
    Last Post: 10-13-2006, 09:31 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •