Robots.txt Help

Can anyone take a look at my robots.txt?

I want to stop spiders from viewing the userprofile. It seems liike stupid google bots enjoy crawling on the user profile only.

Thanks

HTML Code:

User-agent: *
Disallow: /forums/ajax.php
Disallow: /forums/attachment.php
Disallow: /forums/calendar.php
Disallow: /forums/cron.php
Disallow: /forums/editpost.php
Disallow: /forums/global.php
Disallow: /forums/image.php
Disallow: /forums/inlinemod.php
Disallow: /forums/joinrequests.php
Disallow: /forums/login.php
Disallow: /forums/member.php
Disallow: /forums/memberlist.php
Disallow: /forums/misc.php
Disallow: /forums/moderator.php
Disallow: /forums/newattachment.php
Disallow: /forums/newreply.php
Disallow: /forums/newthread.php
Disallow: /forums/online.php
Disallow: /forums/poll.php
Disallow: /forums/postings.php
Disallow: /forums/printthread.php
Disallow: /forums/private.php
Disallow: /forums/profile.php
Disallow: /forums/register.php
Disallow: /forums/report.php
Disallow: /forums/reputation.php
Disallow: /forums/search.php
Disallow: /forums/sendmessage.php
Disallow: /forums/showgroups.php
Disallow: /forums/subscription.php
Disallow: /forums/threadrate.php
Disallow: /forums/usercp.php
Disallow: /forums/usernote.php

Do you have robots.txt file at your forum root ? If yes just remove /forums/ from each file.

This is the file located in my public_html.

I need to remove forums? How do I stop spiders from viewing user profile?

add robots.txt file under your vb root folder with the following content

Code:

User-agent: *
Disallow: /ajax.php
Disallow: /attachment.php
Disallow: /calendar.php
Disallow: /cron.php
Disallow: /editpost.php
Disallow: /global.php
Disallow: /image.php
Disallow: /inlinemod.php
Disallow: /joinrequests.php
Disallow: /login.php
Disallow: /member.php
Disallow: /memberlist.php
Disallow: /misc.php
Disallow: /moderator.php
Disallow: /newattachment.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /poll.php
Disallow: /postings.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /profile.php
Disallow: /register.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /search.php
Disallow: /sendmessage.php
Disallow: /showgroups.php
Disallow: /subscription.php
Disallow: /threadrate.php
Disallow: /usercp.php
Disallow: /usernote.php

I have it under my public_html. Isnt this the same thing?

The robots.txt file should always be located at your domain root. In your case, soletrader, that is public_html. Your path statements should include the direct path FROM your domain root. I checked the location of your robots.txt file and it is correct.

The problem you are experiencing is because you have rewritten the url to your user profiles. It is no longer /forums/profile.php. Your own profile is my example: /forums/1-soultrader/ You may have a tough time keeping spiders away from profiles like that. You may want to customize the rewrite rule for Member Profile URL Format to something like profiles/[user_id]-[user_name].html and make the robots.txt entry read something like Disallow: /forums/profile/. That will keep spiders out of all profiles.

On a side note, I prefer to let my member profiles be spidered. They have a lot of good keywords in their profiles.

Jim

Thanks eJM. I needed that info

...and if you don't want your user profiles to be indexed, you should really turn off the rewriting for them... That will save you some CPU load, and make it easier to block them via ROBOTS.TXT.