robots.txt

Does anyone have any recommendations on robots.txt file?

I would rather be safe than sorry

Thanks for your thoughts

assuming your forums are in / (root)

add /forums/ to the beginning if you are in /forums

also note, i block showpost, the archive, and print versions.

Code:

User-agent: *

Disallow: /ajax.php
Disallow: /attachment.php
Disallow: /calendar.php
Disallow: /cron.php
Disallow: /editpost.php
Disallow: /global.php
Disallow: /image.php
Disallow: /faq.php
Disallow: /inlinemod.php
Disallow: /joinrequests.php
Disallow: /login.php
Disallow: /misc.php
Disallow: /moderator.php
Disallow: /newattachment.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /poll.php
Disallow: /postings.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /profile.php
Disallow: /register.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /search.php
Disallow: /sendmessage.php
Disallow: /payments.php
Disallow: /showpost.php
Disallow: /showgroups.php
Disallow: /spiders.php
Disallow: /subscription.php
Disallow: /threadrate.php
Disallow: /usercp.php
Disallow: /usernote.php
Disallow: /admincp/
Disallow: /cgi-bin/
Disallow: /includes/
Disallow: /install/
Disallow: /modcp/
Disallow: /customavatars/
Disallow: /xmlrpc.php
Disallow: /archive/
Disallow: /sitemap/


#use the following if you have photopost:

Disallow: /gallery/member.php
Disallow: /gallery/showgallery.php?cat=
Disallow: /gallery/showgallery.php?limit=
Disallow: /gallery/comshow.php
Disallow: /gallery/search.php
Disallow: /gallery/*search.php
Disallow: /gallery/showframe.php
Disallow: /gallery/slideshow.php
Disallow: /gallery/*limit
Disallow: /gallery/*ppuser
Disallow: /gallery/*views
Disallow: /gallery/*date
Disallow: /gallery/*recent
Disallow: /gallery/*in


Sitemap: http://typeyourdomainhere.com/sitemap_index.xml.gz

Thanks alot, this will be a good start.

should I add this line as well?

Sitemap: http://typeyourdomainhere.com/sitemap_index.xml.gz

Sitemap: http://typeyourdomainhere.com/sitemap_index.xml.gz

that's be good in the robot.txt file?

Originally Posted by Sysop

that's be good in the robot.txt file?

Yes: Robots.txt now support the sitemap file