is it good idea to stop crawling some forums?

[Seleno]

i disallow some forums in robots.txt but googlebot and yahoo still crawling them !!!
what can i do?

[Seleno]

????

[amenadiel]

Hmmm why don't just make those forums for registered users only?

[Seleno]

its good idea but if i can stop it by robots will be better
i blocked them in robots but google still crawling them!

[Seleno]

any more help about this plz?
coz closing forums not good for my ads !

[Seleno]

Originally Posted by Joe Ward

That should work. It will only effect content found below /vb/f38/ folder.

Joe Ward this didn;t work with me coz threads not found below /f38 or below forum id, they are below /vb
google only stop crawling the forum,but didn't stop crawling its threads !

[Joe Ward]

Why didn't you rewrite your threads within /f38?

That would have created a better hierarchical structure.

[amenadiel]

Originally Posted by Seleno

Joe Ward this didn;t work with me coz threads not found below /f38 or below forum id, they are below /vb
google only stop crawling the forum,but didn't stop crawling its threads !

ok, in case you don't want to rebuild your rewrite structure, there is another way. It won't stop searchbots to crawl your threads, but they won't index any of them.

In your headinclude template add this:

<if condition="$foruminfo[forumid]==38">
<meta name="robots" content="noindex, nofollow" /> 
</if>

use it with caution and make sure it isn't affecting your relevant forums, for it might get your whole site deindexed.

[Jason_A]

I have thought about doing this, but never have.

Joe, I run a tech site, and the majority of the forums I have are for computer related questions. However, I have a few forums that are nothing to do with computers, such as; Chit Chat, Announcements (milestones, birthdays, ect), Fun and Games. Would it be beneficial to block these kind of forums, or is it best keeping until your a little bigger, so you can get more indexed pages?

[Joe Ward]

Originally Posted by Jason Amison

I have thought about doing this, but never have.

Joe, I run a tech site, and the majority of the forums I have are for computer related questions. However, I have a few forums that are nothing to do with computers, such as; Chit Chat, Announcements (milestones, birthdays, ect), Fun and Games. Would it be beneficial to block these kind of forums, or is it best keeping until your a little bigger, so you can get more indexed pages?

You should check to see if they are generating any traffic. If so, they may have some value to you (as a part of the long tail effect).

However, when off-topic forums are created, especially onces that have content either completely non-valuable to web searchers, or very distant from your forum theme, then it's reasonable to block that content, IMO.

You see, the long tail effect occurs because of a huge wealth of content. However, that wealth of content should still be related to your theme for best results. It will also help ensure that the traffic you do get is more likely to stick around.

If I were running a forum with off topic sub-forums, I think I would consider making them for registered users only so they don't show up for search engines.

Again - check their current status. Maybe they are delivering some traffic for you.

If not, then what they are doing is simply making your search engine crawling cycle less efficient. That is, the bots are spending time munching on content that has no value, when they could be munching on your topic-focused content capable of delivering valuable long tail traffic.

This is also the reason, I would go on record as saying that a proactive moderating team should "fix bad thread titles". If you have a thread with good content, but a bad title, it can be impacted. Change it to give it a better chance of ranking.

[Seleno]

Originally Posted by amenadiel

In your headinclude template add this:

<if condition="$foruminfo[forumid]==38">
<meta name="robots" content="noindex, nofollow" /> 
</if>

use it with caution and make sure it isn't affecting your relevant forums, for it might get your whole site deindexed.

this will not solve the problem, google may stop crawling f38 only,but it will keep crawling its thread
i can close these forums and make them for members only,but in this case i will earn less money!

[amenadiel]

Originally Posted by Seleno

this will not solve the problem, google may stop crawling f38 only,but it will keep crawling its thread
i can close these forums and make them for members only,but in this case i will earn less money!

nope, that condition will trigger when you're inside a thread whose forum is 38.

Look at my forum 38: http://www.chw.net/foro/pda-y-moviles-f38/

note that in the source code I added a comment:<!--///// You are still in forum 38 /////-->

take a look on any thread in that forum, for example: http://www.chw.net/foro/rom-update-p...d-t154230.html

look at the source code, what do you see?

[Seleno]

my forum is different
like this: /vb/f38/
and the threads like this: /vb/t2000.html
so google stopped crawling only f38 but not its threads!

[amenadiel]

Remember that I told you:

Originally Posted by amenadiel

It won't stop searchbots to crawl your threads, but they won't index any of them.

there's nothing stopping googlebot from entering your threads, it's just not allowed to index them.

[Jason_A]

Thanks for that, Joe.

So, if I was blocking the announcements forum -- I'd just add the following to my robots.txt:

Dissallow: /forum/announcements/

?