is it good idea to stop crawling some forums?

[Seleno]

Originally Posted by amenadiel

Remember that I told you:



there's nothing stopping googlebot from entering your threads, it's just not allowed to index them.

yeah i understand that,but Googlebot will index the better content faster !!!!
is this right?
and will crawl the usfull threads faster and better also
what do you think?

[amenadiel]

yeah, it should find the robots meta at the beggining of the thread and then go like: "nothing to see here, let's move on"

[Seleno]

thanx amenadiel
my robot file on the index only and not in forum
should i add the robot meta to forum also?

[amenadiel]

robots.txt belongs to your root and you should leave it there.

The robot meta tag is a different thing and you should only apply it to those subforums you don't want indexed. Again, don't ever use that tag on your "good content" subforums for you will get them deindexed.

[Seleno]

well i will put every thing normal and will allow google to index all
coz if googlebot work faster for the other threads then i'm the loser!
and after closing some forums i didn't notice that googlebot is crawling faster really
its the same,it was crawling 25000 pages/daily and now about 14000