Is my robots.txt really working?

Can someone help me to check and see if my robots.txt is really working? I have sett disallow for the forum vbcms comments but it is still being indexed by Google bot.

robots.txt is a suggestion only. Bots will still do what they want.

Okay Brian, but I have a very annoying problem here. Google bot index my articles from the vBCMS comments and not from the real articles in the CMS section. The vBCMS comments only includes a link to the real article and no content really. What can I do about this?

Yell at vb to make their system function properly? :X

The cms comments don't exist outside of the cms comments forum. The display on the cms itself is just a 'copy' of the post from the forum. ie, the cms doesn't have it's own comment engine, so they hacked the current nonsense together.

If you block the forum from guests, then the cms doesn't get any replies on it either. It's just poor design from step 1.

I was told I should block the vBCMS comments for guests and non registered people. Maybe this help? The articles are the most important part of my board so they really should be indexed correctly by Google and other search engines.

if you do that, your articles will have no comments displayed on them to bots/guests at all.

Really? That sucks! No solution then I guess.

Yeah, it needs to be recoded by vb :(

I see this as a bug and will report it as such.

Vote for it: Login Required - vBulletin JIRA.

They already marked it as 'resolved'.

Outstanding.

vb, if you're reading this, the display method of cms comments needs to be fixed. You may not think it's 'broken', and it's not... it's just, well, stupid for a lack of a better word in the way it currently displays in 2 areas.

If it can't be re-coded, the vBCMS comments default forum needs to have special permissions applied to it so that it is not viewable when called via forumdisplay or showthread, rather only from content.php.

Yes here is their reply:

CMS comments are actually just forum posts displayed in the CMS. If you disallow guests from accessing the vBCms Comments forum you're disallowing them from viewing the posts in that forum -> you're disallowing them to view CMS comments.
As long as the way CMS comments are stored won't change, this behaviour is what it's supposed to happen due to how this is currently designed.

yup, current design = fail

This is DISGUSTING!!!! Not only do they design a CMS that does absolutely POORLY in SEO with workflow/features that would have been cutting edge in 1999....... Now they are physically presented a flaw in SEO of their system and essential mark it, "Don't Care".

This is a TOTAL spit in the face.

The people making these decisions obviously don't make a living off of their communities. They are formally declaring that they do not care about my traffic and my livelihood. Without high search engine rankings I die. It's as simple as that.

I'm sure they are busy guys and have to pick their priorities, but to not care if we run successful forums is just insane to me.

I've waited over a year for my rankings to be what they once were with Wordpress as my articles. Man, I regret switching SOOO bad. I wonder if it would be smart to switch back to Wordress. I'd take another short term hit in rankings, but I've got to think that my site would recover faster in Google than vBulletin would.

What a disgusting company! That will have to be the last vBulletin product I ever buy.

Brandon

Did you switch back to WP? How did that go?