Robots.txt like vBSEO

**zems** · 06-08-2009 01:48 AM

Is it ok to use same robots like vBSEO ? I still don't have robots.txt file and I want to make it but I am not sure what to put inside.

Thank you

**Michael Biddle** · 06-08-2009 03:22 AM

What is your rewrite structure?

**zems** · 06-08-2009 03:28 AM

What informations exactly do you need?

**Michael Biddle** · 06-08-2009 03:33 AM

What do your URLs look like? Give us a link to your board.

**zems** · 06-08-2009 03:37 AM

Download Rapidshare Serbia Srbija Filmovi Muzika Igrice

**Michael Biddle** · 06-08-2009 04:08 AM

You could use ours without blocking your pages.

**zems** · 06-08-2009 05:36 AM

Thank you Michael

**redhawk89** · 07-05-2009 05:39 PM

I'm also wondering if I could use VBSEO's robots file... my forum is at SaskForum

**Michael Biddle** · 07-05-2009 05:46 PM

Since your rewrites do not include html or php pages, you can add a robots file containing:

Code:

User-agent: *
Disallow: /*.php
Disallow: /*.html

**Norkus** · 07-12-2009 07:04 PM

Can I ask the same question please?

Hitler Youth Forum

(Don't worry, it's a collector/researcher forum)

I disallowed php, html and htm and then watched my googlbot activity drop to almost nothing from frantic only a few hours earlier. I removed disallow php and back they came in force. Okay, not necessarily linked but it was definitely a worry when I saw it.

Can I safely disallow php?

**marco1** · 07-12-2009 07:08 PM

Originally Posted by Norkus

Can I ask the same question please?

Hitler Youth Forum

(Don't worry, it's a collector/researcher forum)

I disallowed php, html and htm and then watched my googlbot activity drop to almost nothing from frantic only a few hours earlier. I removed disallow php and back they came in force. Okay, not necessarily linked but it was definitely a worry when I saw it.

Can I safely disallow php?

Check this post : Follow-up critique please

**Michael Biddle** · 07-12-2009 08:50 PM

Originally Posted by Norkus

Can I ask the same question please?

Hitler Youth Forum

(Don't worry, it's a collector/researcher forum)

I disallowed php, html and htm and then watched my googlbot activity drop to almost nothing from frantic only a few hours earlier. I removed disallow php and back they came in force. Okay, not necessarily linked but it was definitely a worry when I saw it.

Can I safely disallow php?

You can use the same one as well that I posted. However, you will need to add the forum directory to the beginning.

**Norkus** · 07-13-2009 07:08 AM

Thanks Michael. So just the one line then for php? I don't need to list all php locations seperately in order to disallow them?

**Michael Biddle** · 07-13-2009 01:21 PM

Correct. The *.php means everything with a .php extension.

**Norkus** · 07-13-2009 01:31 PM

Okay, slowly understanding this

How does this impact on showthread.php and forumdisplay.php? I want these to be seen of course but with a robots.txt containing
disallow: /forum/*.php what will the robots be indexing? This is the part I don't understand.

I'm blocking html at the moment as all of my urls are now in the 't17' 'p45' etc format and I have a large robots.txt blocking many .php sections but not showthread.php and forumdisplay.php.

If I could finally get my head around this I'd be a happy man. Knowing that my sitemap and robots.txt and of course my vbseo settings are a match for each other would probably be sufficient reason for a party..

Robots.txt like vBSEO

Robots.txt like vBSEO

Similar Threads

robots.txt managed by vBSEO

Redirecting /forums/robots.txt to /robots.txt - Is it good?

robots.txt and vBSEO

robots.txt und vbseo

robots.txt with vbSEO?