Please review my robots.txt file...

I've went through my ftp directory and created my robots.txt file and would appreciate it if I could get some confirmation if I have this right. I don't want to mess up my vBSEO purchase right off the bat.

I'm not sure about the #Disallow /index.php Should I leave this as is, remove the # symbol, or remove it all together?

Also... do I need to enter in duplicates with the ? afterwards since I have vBAdvanced?

We have a couple of hidden forums as well and a couple one forum that requires registration and login to view and download files. Do I need to do anything with it?

Many thanks!

For domain: Home Theater Shack

# All robots will spider the domain

# Disallow directory
User-agent: *
Disallow: /admincp/
Disallow: /ajax.php
Disallow: /articlebot/
Disallow: /attachment.php
Disallow: /attachments/
Disallow: /clientscript/
Disallow: /cpstyles/
Disallow: /customavatars/
Disallow: /customprofilepics/
Disallow: /calendar.php
Disallow: /cgi-bin/
Disallow: /cron.php
Disallow: /editpost.php
Disallow: /external.php
Disallow: /favicon.ico
Disallow: /frm_attach
Disallow: /image.php
Disallow: /images/
Disallow: /includes/
#Disallow: /index.php
Disallow: /inlinemod.php
Disallow: /install/
Disallow: /joinrequests.php
Disallow: /login.php
Disallow: /member.php?
Disallow: /memberlist.php
Disallow: /misc.php
Disallow: /modcp/
Disallow: /moderator.php
Disallow: /modules/
Disallow: /newattachment.php
Disallow: /newreply.php
Disallow: /newrs.cal
Disallow: /newthread.php
Disallow: /oldindex.html
Disallow: /online.php
Disallow: /payment_gateway.php
Disallow: /payments.php
Disallow: /poll.php
Disallow: /postings.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /profile.php
Disallow: /register.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /search.php
Disallow: /sendmessage.php
Disallow: /showgroups.php
Disallow: /showpost.php
Disallow: /ssm.js
Disallow: /ssmItems.js
Disallow: /subscription.php
Disallow: /subscriptions/
Disallow: /usercp.php
Disallow: /threadrate.php
Disallow: /usercp.php
Disallow: /usernote.php
Disallow: /vbgooglemap.php
Disallow: /vBRadio.php
Disallow: /wizardinstallv3.29.zip
Disallow: /wizardjarv3.29.zip

Anyone? Please?

Hello Sonnie,

you can remove the "#Disallow: /index.php" line.
Also you may want to get indexed these entries:

Disallow: /image.php
Disallow: /showpost.php

Okay... I took out printthread.php and showpost.php and all of that index.php on the advise of Joe Ward.

I'll take out images and image.php too.

What about adding any links with the ? on the end since I have vBAdvanced? Are those necessary since they seem to be duplicate files?

Thank you!

Hello Sonnie,

the robots.txt disallow line will match anything that starts with the string specified.
So, there is no need to enter "page.php?" if you have "page.php" already added.

This may help anyone wanting to validate their robots.txt file

http://tool.motoricerca.info/robots-checker.phtml

I stumbled over this thread some minutes ago.

Is it a good idea seo-wise to exclude those a.m. pages via robots.txt?

Which pages are to be excluded if I use vBSEO?

Isn't it better to get more pages included in the index?

Originally Posted by Lian

I stumbled over this thread some minutes ago.

Is it a good idea seo-wise to exclude those a.m. pages via robots.txt?

Which pages are to be excluded if I use vBSEO?

Isn't it better to get more pages included in the index?

Why will you want google to focus on newthread.php or usercp.php . By disallowing this kind of pages bots can focus on your thread pages.

Ok, that's a good point.

Just for the record... here is my latest:

Code:

# For domain: http://www.hometheatershack.com

# All robots will spider the domain

# Disallow directory
User-agent: *
Disallow: /forums/admincp/
Disallow: /forums/ajax.php
Disallow: /forums/articlebot/
Disallow: /attachment.php
Disallow: /attachments/ 
Disallow: /forums/clientscript/
Disallow: /forums/cpstyles/
Disallow: /forums/customavatars/
Disallow: /forums/customprofilepics/
Disallow: /forums/calendar.php
Disallow: /cgi-bin/
Disallow: /forums/cron.php
Disallow: /display.html
Disallow: /forums/editpost.php
Disallow: /forums/external.php
Disallow: /frm_attach
Disallow: /forums/includes/
Disallow: /forums/inlinemod.php 
Disallow: /forums/install/
Disallow: /forums/ipinfo.php
Disallow: /forums/joinrequests.php
Disallow: /forums/login.php
Disallow: /forums/markers.xml 
Disallow: /forums/member.php 
Disallow: /forums/memberlist.php
Disallow: /forums/misc.php
Disallow: /forums/modcp/
Disallow: /forums/moderator.php
Disallow: /ModeratorOptions.pdf
Disallow: /forums/modules/
Disallow: /forums/newattachment.php
Disallow: /forums/newreply.php 
Disallow: /newrsanalog.cal
Disallow: /newrsdigital.cal
Disallow: /forums/newthread.php
Disallow: /oldrsanalog.cal
Disallow: /oldindex.html
Disallow: /forums/online.php
Disallow: /forums/payment_gateway.php
Disallow: /forums/payments.php
Disallow: /forums/poll.php
Disallow: /forums/postings.php 
Disallow: /forums/private.php 
Disallow: /forums/profile.php 
Disallow: /forums/register.php
Disallow: /forums/report.php
Disallow: /forums/reputation.php
Disallow: /forums/search.php
Disallow: /forums/sendmessage.php
Disallow: /forums/showgroups.php
Disallow: /ssm.js
Disallow: /ssmItems.js
Disallow: /forums/subscription.php
Disallow: /subscriptions/
Disallow: /test.html
Disallow: /forums/test.php
Disallow: /forums/threadrate.php
Disallow: /forums/usercp.php
Disallow: /forums/usernote.php
Disallow: /forums/vbgooglemapme.php
Disallow: /forums/vBRadio.php
Disallow: /forums/vbseocp.php
Disallow: /roomeq/wizardinstallv3.29.zip
Disallow: /roomeq/wizardjarv3.29.zip
Disallow: /roomeq/oldindex.html

what about disallowing *.php? So google would only index everything that was made static via vBSEO or am I wrong?

Can I safely disable things for example like /clientscripts/ or /modules/ without preventing to index everything then?

And what about excluding /images/?

If anyone thinks that the Firefox extension "fasterfox" is causing a problem, you might consider adding this

User-agent: Fasterfox
Disallow: /