How can Google crawl a Private Forum!?

[TMF]

How can Google crawl a private forum?

I know of a forum that has been crawled for over a year now and 99% of their forum is private, you have to register in order to view their content. But you can still see some cached content form google....They get around 20-30 new signups a day!

How can I do this?

I also know that phpbb has a set usergroup for all bots, but does vbulletin have the same?

[briansol]

it can't.... unless they are using a black hat bot sniffer technique.

long story short: if you want your content indexed, leave it open to the public.

[exceem]

I thought you could give google a username/password to access restricted webpages, or was that just for adsense?

[nfn]

that's just for adsense.

[faquick]

as briansol said, they are detecting the bots by their useragent, and letting them in

IMHO this is not the right thing to do, because:

a. people will be able to bypass required registration by just watching the cached copy
b. it would annoy possible new users, if they are forced to register they can just switch to the next search result and ignore your forum

my 2 cents

[Mert Gökçeimam]

as briansol said, they are detecting the bots by their useragent, and letting them in

IMHO this is not the right thing to do, because:

a. people will be able to bypass required registration by just watching the cached copy
b. it would annoy possible new users, if they are forced to register they can just switch to the next search result and ignore your forum

my 2 cents

Third and most important is , if someone reports your site to SE's your site will be dropped from index.

[faquick]

you're right Mert I was thinking only about the people/usability part, but you pointed out a very important fact

thanks