User profile robots.txt

I want to disable user profiles altogether from search engines, but they continue to view them constantly anyways. This is my robots.txt


User-Agent: Googlebot-Image
Disallow: /
User-agent: *
Disallow: /attachment.php
Disallow: /faq.php
Disallow: /newattachment.php
Disallow: /avatar.php
Disallow: /forumdisplay.php?f=23
Disallow: /forumdisplay.php?f=32
Disallow: /forumdisplay.php?f=33
Disallow: /forumdisplay.php?f=39
Disallow: /members
Disallow: /editpost.php
Disallow: /login.php
Disallow: /member.php
Disallow: /member.php?
Disallow: member.php?u=
Disallow: /member2.php
Disallow: /misc.php
Disallow: /moderator.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /calendar.php
Disallow: /online.php
Disallow: /poll.php
Disallow: /postings.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /private2.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /search.php
Disallow: /sendtofriend.php
Disallow: /showgroups.php
Disallow: /showpost.php
Disallow: /threadrate.php
Disallow: /usercp.php
Disallow: /usernote.php
Disallow: /sendmessage.php
Disallow: /register.php
Disallow: /subscription.php
Disallow: /admincp/
Disallow: /archive/
Disallow: /clientscript/
Disallow: /cpstyles/
Disallow: /customavatars/
Disallow: /images/
Disallow: /includes/
Disallow: /install/
Disallow: /modcp/
Disallow: /subscriptions/

Did you remove member.php files from your sitemaps ? Check vBSEO Google/Yahoo Sitemap Generator settings and make sure didn't add member profiles to your sitemaps.

Originally Posted by Mert Gökçeimam

Did you remove member.php files from your sitemaps ? Check vBSEO Google/Yahoo Sitemap Generator settings and make sure didn't add member profiles to your sitemaps.

I dont have sitemaps.

You should be running vBSEO Sitemap for best results.

Your ROBOTS.TXT should be fine. Just disable rewrites for user profiles in your vBSEO control panel. That will cause them to use the default vBulletin URLs, which will be blocked by your member.php entry in your ROBOTS.TXT.

Add /members/ to robots.txt as they are re-written URLs.

member.php only excludes the old URLs.

If you aren't allowing search engines to index them, there's no need to rewrite member profiles. It's just using extra CPU that isn't necessary.

Isnt rewriting profiles enabled by default?

How do you disable it?

Under the URL section in the vBSEO Control Panel, click the + (plus sign) to open up all of the options for customizing.

Dave:
Out of interest, which spiders are crawling profiles ?

I've noticed google doesn't however the yahoo crawler does and it shows as a stop "No Permissions" error in WOL. I've all but eliminated this for the yahoo crawler by placing conditionals around certain code in a few templates. Experimental.

All spiders will spider them unless you block the via robots.txt.

Why bother adding code when one line in robots.txt stops them 100%.

Are you certain about that. What if they follow a direct link from any page anywhere on the forum ? There are indeed many that link direct to profiles.


For yahoo it makes no difference if there is a rel="nofollow"