I'm looking for thoughts/opinions on what should be in robots.txt for my forum. At the moment, I have:
User-agent: *
Disallow: /acp/
Disallow: /archive/
Disallow: /cgi-bin/
Disallow: /clientscript/
Disallow: /cpsytles/
Disallow: /customavatars/
Disallow: /customprofilepics/
Disallow: /images/
Disallow: /includes/
Disallow: /install/
Disallow: /mcp/
Disallow: /signaturepics/
Disallow: /vbseo_sitemap/
Disallow: ajax.php
Disallow: album.php
Disallow: announcement.php
Disallow: attachment.php
Disallow: calendar.php
Disallow: clear.gif
Disallow: converse.php
Disallow: cron.php
Disallow: editpost.php
Disallow: faq.php
Disallow: forumdisplay.php
Disallow: global.php
Disallow: group.php
Disallow: group_inlinemod.php
Disallow: image.php
Disallow: infraction.php
Disallow: inlinemod.php
Disallow: joinrequests.php
Disallow: login.php
Disallow: member.php
Disallow: member_inlinemod.php
Disallow: memberlist.php
Disallow: misc.php
Disallow: moderation.php
Disallow: moderator.php
Disallow: newattachment.php
Disallow: newreply.php
Disallow: newthread.php
Disallow: online.php
Disallow: payment_gateway.php
Disallow: payments.php
Disallow: picture.php
Disallow: picture_inlinemod.php
Disallow: picturecomment.php
Disallow: poll.php
Disallow: posthistory.php
Disallow: postings.php
Disallow: printthread.php
Disallow: private.php
Disallow: profile.php
Disallow: register.php
Disallow: report.php
Disallow: reputation.php
Disallow: search.php
Disallow: sendmessage.php
Disallow: showgroups.php
Disallow: showpost.php
Disallow: showthread.php
Disallow: subscription.php
Disallow: tags.php
Disallow: threadrate.php
Disallow: threadtag.php
Disallow: usercp.php
Disallow: usernote.php
Disallow: vbseo.php
Disallow: vbseocpform.php
Disallow: visitormessage.php


LinkBack URL
About LinkBacks





Reply With Quote
I had actually not realized until you mentioned it that I included my mod and admin directories, oops. I'll remove those. I have everything rewritten now, so it would probably best to:

