Do i need a robots.txt?

Originally Posted by Hendricius

That guy is forbidding the archive? That sounds very stupid to me...

Actually I think having the archive enabled at all is the 'stupid' part.

Hmm somehow true. I just read a thread about how useless it was under the double content aspect.

Could you guys share your robots.txt? I would like to exclude some pages for my forum, it is located in /forum. Do I have to put the robots.txt in the /forum? Or do I have to put it in the root and then disallow /forum/... ?

Thanks.

Add, thanks to briansol robots.txt. I am now using the following:

User-agent: *

Disallow: /forum/ajax.php
Disallow: /forum/attachment.php
Disallow: /forum/calendar.php
Disallow: /forum/cron.php
Disallow: /forum/editpost.php
Disallow: /forum/global.php
Disallow: /forum/image.php
Disallow: /forum/faq.php
Disallow: /forum/inlinemod.php
Disallow: /forum/joinrequests.php
Disallow: /forum/login.php
Disallow: /forum/misc.php
Disallow: /forum/moderator.php
Disallow: /forum/newattachment.php
Disallow: /forum/newreply.php
Disallow: /forum/newthread.php
Disallow: /forum/online.php
Disallow: /forum/poll.php
Disallow: /forum/postings.php
Disallow: /forum/printthread.php
Disallow: /forum/private.php
Disallow: /forum/profile.php
Disallow: /forum/register.php
Disallow: /forum/report.php
Disallow: /forum/reputation.php
Disallow: /forum/search.php
Disallow: /forum/sendmessage.php
Disallow: /forum/payments.php
Disallow: /forum/showpost.php
Disallow: /forum/showgroups.php
Disallow: /forum/spiders.php
Disallow: /forum/subscription.php
Disallow: /forum/threadrate.php
Disallow: /forum/usercp.php
Disallow: /forum/usernote.php
Disallow: /forum/admincp/
Disallow: /forum/cgi-bin/
Disallow: /forum/includes/
Disallow: /forum/install/
Disallow: /forum/modcp/
Disallow: /forum/customavatars/
Disallow: /forum/xmlrpc.php
Disallow: /forum/archive/
Disallow: /forum/sitemap/

http://www.d3scene.com/forum/sitemap_index.xml.gz

Placed it in my domains root. Is that okay? Thanks.

Hendricius, not really: remove space under "User-agent: *" and above "http://www.d3scene.com/forum/sitemap_index.xml.gz", and change "http://www.d3scene.com/forum/sitemap_index.xml.gz" to "Sitemap: http://www.d3scene.com/forum/sitemap_index.xml.gz"

spaces/line breaks don't matter.

but yes, you're missing the sitemap: directive as suggested above.


Robots.txt ONLY works in root, and all file locations are relative to that point, so yes, you need the /forums/ on yours.


as for the archive, you can see in my file that i have it robots'ed out, and it is completely off in the VB control panel setting for it. my archive does not even exist.

Ok forgive me for being stupid, were do I turn off archive.

to forbid the archive:

Disallow: /forum/archive/

Thanks for the tips .

The real turn-off is in the vbulletin ACP, general options, search engine friendly archive setting group from the select box.

the disallow simply stops spidering, but doesn't disable it.

Originally Posted by briansol

The real turn-off is in the vbulletin ACP, general options, search engine friendly archive setting group from the select box.

the disallow simply stops spidering, but doesn't disable it.

So should I turn it off this way?

allso here is my robots.txt

User-agent: *
Disallow: /forums/ajax.php
Disallow: /forums/attachment.php
Disallow: /forums/calendar.php
Disallow: /forums/cron.php
Disallow: /forums/editpost.php
Disallow: /forums/global.php
Disallow: /forums/image.php
Disallow: /forums/faq.php
Disallow: /forums/inlinemod.php
Disallow: /forums/joinrequests.php
Disallow: /forums/login.php
Disallow: /forums/misc.php
Disallow: /forums/moderator.php
Disallow: /forums/newattachment.php
Disallow: /forums/newreply.php
Disallow: /forums/newthread.php
Disallow: /forums/online.php
Disallow: /forums/poll.php
Disallow: /forums/postings.php
Disallow: /forums/printthread.php
Disallow: /forums/private.php
Disallow: /forums/profile.php
Disallow: /forums/register.php
Disallow: /forums/report.php
Disallow: /forums/reputation.php
Disallow: /forums/search.php
Disallow: /forums/sendmessage.php
Disallow: /forums/payments.php
Disallow: /forums/showpost.php
Disallow: /forums/showgroups.php
Disallow: /forums/spiders.php
Disallow: /forums/subscription.php
Disallow: /forums/threadrate.php
Disallow: /forums/usercp.php
Disallow: /forums/usernote.php
Disallow: /forums/admincp/
Disallow: /cgi-bin/
Disallow: /forums/includes/
Disallow: /forums/install/
Disallow: /forums/modcp/
Disallow: /forums/customavatars/
Disallow: /forums/xmlrpc.php
Disallow: /forums/archive/
Disallow: /forums/sitemap/
Disallow: /forums/gallery/member.php
Disallow: /forums/gallery/showgallery.php?cat=
Disallow: /forums/gallery/showgallery.php?limit=
Disallow: /forums/gallery/comshow.php
Disallow: /forums/gallery/search.php
Disallow: /forums/gallery/*search.php
Disallow: /forums/gallery/showframe.php
Disallow: /forums/gallery/slideshow.php
Disallow: /forums/gallery/*limit
Disallow: /forums/gallery/*ppuser
Disallow: /forums/gallery/*views
Disallow: /forums/gallery/*date
Disallow: /forums/gallery/*recent
Disallow: /forums/gallery/*in
Sitemap: http://www.taskforcealliance.com/sitemap_index.xml.gz

does that look ok? anything I can add for VBAlinks Directory and Downloads II, oh and I have TV/Radio hack installed.

Advice appreciated