Robots.txt like vBSEO

Is it ok to use same robots like vBSEO ? I still don't have robots.txt file and I want to make it but I am not sure what to put inside.

Thank you

What is your rewrite structure?

What informations exactly do you need?

What do your URLs look like? Give us a link to your board.

Download Rapidshare Serbia Srbija Filmovi Muzika Igrice

You could use ours without blocking your pages.

Thank you Michael

I'm also wondering if I could use VBSEO's robots file... my forum is at SaskForum

Since your rewrites do not include html or php pages, you can add a robots file containing:

Code:

User-agent: *
Disallow: /*.php
Disallow: /*.html

Can I ask the same question please?

Hitler Youth Forum

(Don't worry, it's a collector/researcher forum)

I disallowed php, html and htm and then watched my googlbot activity drop to almost nothing from frantic only a few hours earlier. I removed disallow php and back they came in force. Okay, not necessarily linked but it was definitely a worry when I saw it.

Can I safely disallow php?

Originally Posted by Norkus

Can I ask the same question please?

Hitler Youth Forum

(Don't worry, it's a collector/researcher forum)

I disallowed php, html and htm and then watched my googlbot activity drop to almost nothing from frantic only a few hours earlier. I removed disallow php and back they came in force. Okay, not necessarily linked but it was definitely a worry when I saw it.

Can I safely disallow php?

Check this post : Follow-up critique please

Originally Posted by Norkus

Can I ask the same question please?

Hitler Youth Forum

(Don't worry, it's a collector/researcher forum)

I disallowed php, html and htm and then watched my googlbot activity drop to almost nothing from frantic only a few hours earlier. I removed disallow php and back they came in force. Okay, not necessarily linked but it was definitely a worry when I saw it.

Can I safely disallow php?

You can use the same one as well that I posted. However, you will need to add the forum directory to the beginning.

Thanks Michael. So just the one line then for php? I don't need to list all php locations seperately in order to disallow them?

Correct. The *.php means everything with a .php extension.

Okay, slowly understanding this

How does this impact on showthread.php and forumdisplay.php? I want these to be seen of course but with a robots.txt containing
disallow: /forum/*.php what will the robots be indexing? This is the part I don't understand.

I'm blocking html at the moment as all of my urls are now in the 't17' 'p45' etc format and I have a large robots.txt blocking many .php sections but not showthread.php and forumdisplay.php.

If I could finally get my head around this I'd be a happy man. Knowing that my sitemap and robots.txt and of course my vbseo settings are a match for each other would probably be sufficient reason for a party..