Help with my robots.txt

Hi,

I know a lot of people simply "Disallow: *.php" in their robots.txt file which is a lot easier than making a huge list of all the individual files, but I'm unable to do it the easy way since I have *.php files that I do not want blocked, like this one for example.

So I just put together my robots.txt, but before I upload it I want to make sure it's alright:

User-Agent: *
Disallow: /forum/archive/
Disallow: /forum/clientscript/
Disallow: /forum/cpstyles/
Disallow: /forum/customavatars/
Disallow: /forum/customgroupicons/
Disallow: /forum/customprofilepics/
Disallow: /forum/images/
Disallow: /forum/includes/
Disallow: /forum/install/
Disallow: /forum/modules/
Disallow: /forum/signaturepics/
Disallow: /forum/tmp/
Disallow: /forum/vbseo_sitemap/
Disallow: /forum/ajax.php
Disallow: /forum/album.php
Disallow: /forum/announcement.php
Disallow: /forum/attachment.php
Disallow: /forum/calendar.php
Disallow: /forum/clear.gif
Disallow: /forum/converse.php
Disallow: /forum/cron.php
Disallow: /forum/editpost.php
Disallow: /forum/external.php
Disallow: /forum/faq.php
Disallow: /forum/favicon.ico
Disallow: /forum/forumdisplay.php
Disallow: /forum/global.php
Disallow: /forum/group.php
Disallow: /forum/group_inlinemod.php
Disallow: /forum/groupsubscription.php
Disallow: /forum/gxboxlive.php
Disallow: /forum/image.php
Disallow: /forum/index.php
Disallow: /forum/infraction.php
Disallow: /forum/inlinemod.php
Disallow: /forum/joinrequests.php
Disallow: /forum/LICENSE
Disallow: /forum/login.php
Disallow: /forum/member.php
Disallow: /forum/member_inlinemod.php
Disallow: /forum/memberlist.php
Disallow: /forum/misc.php
Disallow: /forum/moderation.php
Disallow: /forum/moderator.php
Disallow: /forum/newattachment.php
Disallow: /forum/newreply.php
Disallow: /forum/newthread.php
Disallow: /forum/online.php
Disallow: /forum/payment_gateway.php
Disallow: /forum/payments.php
Disallow: /forum/picture.php
Disallow: /forum/picture_inlinemod.php
Disallow: /forum/picturecomment.php
Disallow: /forum/poll.php
Disallow: /forum/posthistory.php
Disallow: /forum/postings.php
Disallow: /forum/printthread.php
Disallow: /forum/privacy.php
Disallow: /forum/private.php
Disallow: /forum/profile.php
Disallow: /forum/register.php
Disallow: /forum/report.php
Disallow: /forum/reputation.php
Disallow: /forum/search.php
Disallow: /forum/sendmessage.php
Disallow: /forum/showgroups.php
Disallow: /forum/showpost.php
Disallow: /forum/showthread.php
Disallow: /forum/subscription.php
Disallow: /forum/tags.php
Disallow: /forum/threadrate.php
Disallow: /forum/threadtag.php
Disallow: /forum/usercp.php
Disallow: /forum/usernote.php
Disallow: /forum/vbseo.php
Disallow: /forum/vbseocp.php
Disallow: /forum/vbseocpform.php
Disallow: /forum/visitormessage.php
Disallow: /forum/xperience.php

Sitemap: http://www.gamersthrone.com/forum/sitemap_index.xml.g

The lines in red are the ones I'm really not sure about, especially the Privacy and gxboxlive files.

Can someone please confirm before I upload this?

Thank you!

I always just delete the LICENSE file
and I wouldn't disallow bots from my index.php file
and I wouldn't worry about the favoricon
other than that, it looks "okay"
here is mine if you want to compare
http://forum.vbulletinsetup.com/robots.txt

Okay, thanks for the reply. I finally uploaded it

I removed the following:

Disallow: /forum/index.php
Disallow: /forum/privacy.php
Disallow: /forum/gxboxlive.php

...this way there wont be any issues viewing the forum, the privacy statement, or my Xbox Live Leaderboard.

I added the following:

Disallow: /forum/tags

Now it looks like this...

User-Agent: *
Disallow: /forum/archive/
Disallow: /forum/clientscript/
Disallow: /forum/cpstyles/
Disallow: /forum/customavatars/
Disallow: /forum/customgroupicons/
Disallow: /forum/customprofilepics/
Disallow: /forum/images/
Disallow: /forum/includes/
Disallow: /forum/install/
Disallow: /forum/modules/
Disallow: /forum/signaturepics/
Disallow: /forum/tmp/
Disallow: /forum/vbseo_sitemap/
Disallow: /forum/tags
Disallow: /forum/ajax.php
Disallow: /forum/album.php
Disallow: /forum/announcement.php
Disallow: /forum/attachment.php
Disallow: /forum/calendar.php
Disallow: /forum/clear.gif
Disallow: /forum/converse.php
Disallow: /forum/cron.php
Disallow: /forum/editpost.php
Disallow: /forum/external.php
Disallow: /forum/faq.php
Disallow: /forum/favicon.ico
Disallow: /forum/forumdisplay.php
Disallow: /forum/global.php
Disallow: /forum/group.php
Disallow: /forum/group_inlinemod.php
Disallow: /forum/groupsubscription.php
Disallow: /forum/image.php
Disallow: /forum/infraction.php
Disallow: /forum/inlinemod.php
Disallow: /forum/joinrequests.php
Disallow: /forum/login.php
Disallow: /forum/member.php
Disallow: /forum/member_inlinemod.php
Disallow: /forum/memberlist.php
Disallow: /forum/misc.php
Disallow: /forum/moderation.php
Disallow: /forum/moderator.php
Disallow: /forum/newattachment.php
Disallow: /forum/newreply.php
Disallow: /forum/newthread.php
Disallow: /forum/online.php
Disallow: /forum/payment_gateway.php
Disallow: /forum/payments.php
Disallow: /forum/picture.php
Disallow: /forum/picture_inlinemod.php
Disallow: /forum/picturecomment.php
Disallow: /forum/poll.php
Disallow: /forum/posthistory.php
Disallow: /forum/postings.php
Disallow: /forum/printthread.php
Disallow: /forum/private.php
Disallow: /forum/profile.php
Disallow: /forum/register.php
Disallow: /forum/report.php
Disallow: /forum/reputation.php
Disallow: /forum/search.php
Disallow: /forum/sendmessage.php
Disallow: /forum/showgroups.php
Disallow: /forum/showpost.php
Disallow: /forum/showthread.php
Disallow: /forum/subscription.php
Disallow: /forum/tags.php
Disallow: /forum/threadrate.php
Disallow: /forum/threadtag.php
Disallow: /forum/usercp.php
Disallow: /forum/usernote.php
Disallow: /forum/vbseo.php
Disallow: /forum/vbseocp.php
Disallow: /forum/vbseocpform.php
Disallow: /forum/visitormessage.php
Disallow: /forum/xperience.php

Sitemap: http://www.gamersthrone.com/forum/sitemap_index.xml.gz

why not just disallow .php? would save alot of issues..... rather then publicly show the files you wish not be indexed, as some crawling bots may ignore the robot.txt

Originally Posted by Martyn

why not just disallow .php? would save alot of issues..... rather then publicly show the files you wish not be indexed, as some crawling bots may ignore the robot.txt

Because...

Originally Posted by JPetrillo

Hi,

I know a lot of people simply "Disallow: *.php" in their robots.txt file which is a lot easier than making a huge list of all the individual files, but I'm unable to do it the easy way since I have *.php files that I do not want blocked, like this one for example.

...

why not setit up within vbseo so your reviews will be either end with a slash or .html ??

you could make your link

Code:

http://www.gamersthrone.com/forum/reviews/xbox360/ghostbusters/ 
or end with .html
http://www.gamersthrone.com/forum/reviews/xbox360/ghostbusters.html
or even!
http://www.gamersthrone.com/forum/xbox360review-ghostbusters.html

rather then the http://www.gamersthrone.com/forum/xb...ostbusters.php

also you can make it easier on your self by doing it within the template and just calling misc.php rather then creating another .php file and template to every review you do, this would be more sufficient, and think down the line when you have 100 reviews, thats 100 .php files your have in your root folder!

Originally Posted by JPetrillo

I removed the following:

Disallow: /forum/privacy.php

Is there any particular reason that you want privacy.php to be indexed?

Originally Posted by Martyn

why not setit up within vbseo so your reviews will be either end with a slash or .html ??

you could make your link

Code:

http://www.gamersthrone.com/forum/reviews/xbox360/ghostbusters/ 
or end with .html
http://www.gamersthrone.com/forum/reviews/xbox360/ghostbusters.html
or even!
http://www.gamersthrone.com/forum/xbox360review-ghostbusters.html

rather then the GamersThrone: Video Game Forums

also you can make it easier on your self by doing it within the template and just calling misc.php rather then creating another .php file and template to every review you do, this would be more sufficient, and think down the line when you have 100 reviews, thats 100 .php files your have in your root folder!

Wow that's a great idea! How do I setup the URL's like that? I have no idea how to do that...

Originally Posted by Zagis

Is there any particular reason that you want privacy.php to be indexed?

I don't know... I thought it was important? I should block that too?