Temp robots.txt Brand New Forum?

[rmjvol]

I'm about to install my 1st vBulletin with vBSEO.

Wanting to play around with it for a few days before allowing the spirders to get at it. Mess with the url options, seed with some initial content... whatever.

Is there any reason I could not robots.txt the whole thing temporarily?

I'm assuming it's as simple as adding a disallow all line in robots.txt and then removing it once I'm ready to let them in. vBSEO will play nice if I do that or is there some potential conflict with my plan?

Thanks, Ross

[ADM]

Well theres a conflict since some search robots can take up to 30 days to see new rules in your robots.txt and then another 30 days on top of that to check it again.

[rmjvol]

Sorry, I was unclear on my question.

Spiders are not my concern, I'm familiar with what to expect from them. I'm not asking what effect this will have upon spiders or the Search Engine's.

My question worded more effectively (I hope):

Will vbseo function properly if I place this in my robots.txt for a temporary period of time?

User-agent: *
Disallow: /

Thanks, Ross

[Keith Cohen]

Yes, it will work, but as ADM said, it may take a long time before spiders start visiting your site even AFTER you remove the robots.txt. If you want to keep it from being spidered, why not just use .htaccess password protection while you work on it?

[rmjvol]

Thanks.

After the first day or so, I'll have a number of people working on seeding the forum with posts so I'd rather not inconvenience them with a double login.

Plus, it's been my experience that if you have enough links, the major spiders will be knocking on the door regularly in anticipation.

Thanks for your comments, I'll let you know if that's still the case

[Oleg Ignatiuk]

Hi Ross,

you can also temporary disable access to all forums for guests in usergroups permissions setting.

[rmjvol]

ah, that's something I hadn't thought of and do the trick, thanks

[majordude]

Or, turn off your toolbars (Yahoo, Google, A9). They send signals to the spiders.