robot.txt for vbulletin 4

Hi
I want to know should i include robot.txt file or not..... I have created one using some help from the older posts .....is it right

Code:

User-agent: *
Disallow: /cgi-bin/
Disallow: /forum/ajax.php
Disallow: /forum/articlebot/
Disallow: /forum/clientscript/
Disallow: /forum/cpstyles/
Disallow: /forum/customavatars/
Disallow: /forum/customprofilepics/
Disallow: /forum/calendar.php
Disallow: /forum/cron.php
Disallow: /forum/editpost.php
Disallow: /forum/global.inc.php
Disallow: /forum/includes/
Disallow: /forum/inlinemod.php
Disallow: /forum/install/
Disallow: /forum/joinrequests.php
Disallow: /forum/login.php
Disallow: /forum/member.php
Disallow: /forum/memberlist.php
Disallow: /forum/moderator.php
Disallow: /forum/moderatorapplication.php
Disallow: /forum/modules/
Disallow: /forum/newattachment.php
Disallow: /forum/newreply.php
Disallow: /forum/newthread.php
Disallow: /forum/online.php
Disallow: /forum/poll.php
Disallow: /forum/postings.php
Disallow: /forum/private.php
Disallow: /forum/process.php
Disallow: /forum/profile.php
Disallow: /forum/register.php
Disallow: /forum/report.php
Disallow: /forum/reputation.php
Disallow: /forum/search.php
Disallow: /forum/sendmessage.php
Disallow: /forum/showgroups.php
Disallow: /forum/subscription.php
Disallow: /forum/threadrate.php
Disallow: /forum/usercp.php
Disallow: /forum/usernote.php
Disallow: /forum/vbseocp.php
Sitemap: http://www.xxxx.com/forum/sitemap_index.xml.gz

This looks fine for me.

1 more question should i disable vbulletin xml generator as i am using vbseo ..... if i can use both of them than can i use both site map in robot file

Yes, the vBulletin sitemap will not produce vBSEO URLs. You can download our sitemap generator from vbseo.com/downloads. Use 2.6 for vBulletin 4.

Quick question.... I know this thread is a few months old,but... I installed the vbseo sitemap. Where do I enter the password so I can view the sitemaps reports...? I get the login page, but is there a default password? When I imported the xml file everything installed ok, but I guess I missed a step somewhere because I never set up the password.

Also, should I use the above posted robot.txt...?

Hi,

You set the password in the AdminCP -> vBSEO -> Sitemap Settings -> Password...





Your Robots.txt file should be individual to you and your needs but saying that if your site is in the /forums/ folder then there is no harm using the above as a guide.

Ceri

I would recommend removing
Disallow: /forum/install/

from the robots file and physically deleting the entire folder from your server after upgrade/installs.
You never want to 'tell' would be hackers where your private admin/mod/install files are located.

Thanks for getting back to me... :-)

My forum is located in the root directory... would it be better not to have a robot.txt file...?

No, you should have one. simply remove the 'forum/' path in the above to reflect your installation location.

Originally Posted by Brian Cummiskey

No, you should have one. simply remove the 'forum/' path in the above to reflect your installation location.

Thanks! :-)

Does this look good to you....?

User-agent: *
Disallow: /cgi-bin/
Disallow: /ajax.php
Disallow: /articlebot/
Disallow: /clientscript/
Disallow: /cpstyles/
Disallow: /customavatars/
Disallow: /customprofilepics/
Disallow: /calendar.php
Disallow: /cron.php
Disallow: /editpost.php
Disallow: /global.inc.php
Disallow: /includes/
Disallow: /inlinemod.php
Disallow: /joinrequests.php
Disallow: /login.php
Disallow: /member.php
Disallow: /memberlist.php
Disallow: /moderator.php
Disallow: /moderatorapplication.php
Disallow: /modules/
Disallow: /newattachment.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /poll.php
Disallow: /postings.php
Disallow: /private.php
Disallow: /process.php
Disallow: /profile.php
Disallow: /register.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /search.php
Disallow: /sendmessage.php
Disallow: /showgroups.php
Disallow: /subscription.php
Disallow: /threadrate.php
Disallow: /usercp.php
Disallow: /usernote.php
Disallow: /vbseocp.php

That looks fine.