Bot Crawling Question

Do bots such as Google, Yahoo etc crawl private forums?

For example, i have certain sections of my forum that I want people to be registered for, however I do want the bots to pick them up....can this be done??

Thanks

Originally Posted by nymyth

Do bots such as Google, Yahoo etc crawl private forums?

For example, i have certain sections of my forum that I want people to be registered for, however I do want the bots to pick them up....can this be done??

Thanks

Log out of your forums.

Now whatever you can see, thats what search engine bots can see.

You'll need a modification or something to show the bots the data thats just for registered users, however that defeat the purpose as if the bots can see it, they'll cache it. Which means anyone will be able to see it.

hmmmm....thanks for that....

another question: does vbSEO allow bots to crawl PM's ... cuz i saw a bot on WOL and it had "Viewing Conversation" as its location

conversation is NOT pm. conversation is visitor messages.

ah gotcha...thanks