Using nofollow or robots.txt

Should we be using robots.txt to block content we don't want crawled or using nofollow tags for internal links? The official definition, as defined by Google, for nofollow are links that you don't vouch for. Using nofollow for internal links would be sending Google the message that you don't vouch for pages on your own domain, which could lead to a problem if a future search algorithmic change took that into consideration.

So wouldn't it be better to just robots.txt to block these, and no have nofollow applied to internal linkings?

Originally Posted by FightRice

The official definition, as defined by Google, for nofollow are links that you don't vouch for.

source?


Nofollow and removed via robots accomplish two different things.


you can nofollow every link possible to a page, but if someone else posts a link to it else where, it can/will be found and possibly indexed.


robots.txt won't let the page be indexed (in theory) regardless of where it's linked.

As Brian suggests, robots.txt is the better option. It will block the content from being indexed regardless of any link to it discovered.

Things like showpost pages, which I prefer to be removed completely, should not have any active links.

Other pages like member profile pages (which I also prefer to block) usually would not be removed as they are a needed part of the user experience.

So - if you can remove a page completely, that is the best option.

If you cannot remove it completely, but want it skipped by the search engines, robots.txt is the best option.

And if it's a link to a 3rd party site, then either not making them available (guests only perhaps) or adding rel=nofollow is the best option.

Thank you again