Using robots.txt or .htaccess to eliminate double-penalty?

**Kaelon** · 04-02-2009 08:00 PM

Hi there!

Here's my current hosting setup:

companysite.com at /home/company/httpdocs
forumsite.com at /home/company/httpdocs/forumsite
forums located at /home/company/httpdocs/forumsite/forums

So, I think I am getting double-penalized, because even though www.forumsite.com/forums points correctly, the robots/spiders are crawling through companysite.com/forumsite/forums as well.

Question:

Is there a specific robots.txt or .htaccess setting that I can use to ensure that whenever users/bots visit my forumsite subdirectory that they do so only through forumsite.com ?

Thanks!

**Kaelon** · 10-29-2009 08:02 PM

I realize I posted this several months ago, but never had an answer. Does anyone have any advice on this?

**Mert Gökçeimam** · 10-29-2009 08:19 PM

Bots don't visit your forum according to their file path. however if you have multiple forum installations , you should either 301 redirect one folder to another or block the second folder from guest access by .htacess protection.

**Kaelon** · 10-29-2009 08:30 PM

That's not really my situation. I have a single forum installation, and it is located at:

/home/username/website <-- website.com

But, I also have a corporate site, whose documents are kept at:

/home/username <-- username.com

Yet people (both bots and regular users) can access my forumsite by going to username.com/website. I'd like to force users to access the website directory using only one valid URL (website.com).

**Mert Gökçeimam** · 10-29-2009 08:34 PM

In that case you should 301 redirect your addon domain to your main domain in order to avoid duplicate content penalty

**Kaelon** · 10-29-2009 08:46 PM

How would I do that?

**Mert Gökçeimam** · 10-29-2009 08:54 PM

[How to] Setup www.yourdomain.com or yourdomain.com (www vs. non-www) - With .htaccess 301 redirects

**Kaelon** · 11-23-2009 01:46 AM

Now that I have eliminated the double-content situation, is there something that I should do to notify bots? Or will this happen gradually? I'm really hoping that this double penalty gets lifted and I get many more of my pages re-spidered.

**Brian Cummiskey** · 11-23-2009 01:59 AM

It should fix itself in due time. It doesn't happen over-night.
You can always try submitting a reconsideration request:
Requesting reconsideration of your site - Webmasters/Site owners Help

Using robots.txt or .htaccess to eliminate double-penalty?

LinkBack

Thread Tools

Display

Using robots.txt or .htaccess to eliminate double-penalty?

Similar Threads

vBSEO 3.2.0 RC5 > 3.3.0 RC2 Upgrade - double check htaccess syntax

Redirecting /forums/robots.txt to /robots.txt - Is it good?

Penalty for using hyphen in pages

penalty for other URLs pointing to yours

strange penalty (due to server problems?)

Posting Permissions