No Follow rules for spiders

I know there was something in the configuration about not allowing spiders to read profiles. Does that apply to only GoogleBots or other spiders too? I ask because I see the Yahoo spider looking at profiles and another trying to email a member.

What kind of rules can I set up (and where/how) to keep spiders from doing some specific things? I don't want them to try emailing or PMing anyone, or trying to edit or post messages. I have seen all of this already. I'd also like to do a no-follow rule on the fly if I knew the standard format and where to put it.

Thank you.

Not all spiders will recognize nofollow's. You should also use a robots.txt, but there are no guarantees that all spiders will follow those rules either.

You currently have this in your robots.txt. This should be blocking all spiders from everything, and you definately don't want that, so remove these lines:

Code:

User-agent: *
Disallow: /

If you don't want spiders to index member profiles, turn off member profile rewriting in your vBSEO control panel, then add this to robots.txt:

Code:

User-agent: *
Disallow: /community/profile.php

Here's mine, with other examples: http://www.fireblades.org/robots.txt

Is there some advantage in allowing a spider to index the profiles? I'm thinking about the website links which could lead to linkbacks. There could also be other information that might add to my keyword stash.

What about preventing replies, emailing and such?

Thanks. I don't know why I had that block everything rule in there. It obviously wasn't working. I've seen Google, Yahoo and AskJeeves spiders quite a bit the past week or so.

If you turn on the option to add the rel="nofollow" for all external links, you won't have to worry about their home page links.

I have a lot of keywords on my profile pages because of all the profile fields where members choose the brands/models/etc, so I want them to be indexed too. It's up to you...

Well then, why would I want to disallow following external links? Couldn't this help me more than hurt me? I am posting links to my member's sites on the home page and asking them to post links to the forum on their sites. This is good, right? So allowing spiders to follow that is an asset to me, right? Or can I be more specific about what parts of the forum have good external links and what are not?

PS: What the heck is wrong with your server today? Tice now I have had to close my browser or surf away just to be able to do anything here.

Allowing spiders to follow those links is an asset to THEM (your users' sites), but not to you. An excess of outbound links actually hurts you.

Ohhhhh, I see. Hmmm, I didn't know that. I'm going to have to read up on link trading (or whatever they call it).

Thanks for cluing me in.