Possible duplicate content

PageUp · #1 11-02-2005, 02:50 PM

I was wondering what you think about these two scenarios and what could be done about it:

1) A malicious user sends SEs lots of links like:
/forum/thread100-page2234234.html,
/forum/thread100-page3634634.html,
/forum/thread100-page4463461.html etc.
... which all show the same page (the last page in a thread). So SE has a couple of different URLs in its index with the exact same content. OK, you'll say that all these pages are orphaned and will just go to the supplemental index (and hope that this will not harm rankings). But what about this:

2) A malicious user sends Google lots of links like:
/forum/showthread.php?t=45&pp=2,
/forum/showthread.php?t=34&pp=2,
/forum/showthread.php?t=89&pp=2 etc.

... which all rewrite perfectly of course, but they all show a different number of posts per page than default setting. The problem is not only the possible double content penalty on the page itself, but also that SE will follow links to prev- and next pages of a thread and this may also lead to more double content issues (as some links might all show the last page in a thread). And this time, I don't think SEs will consider these URLs orphaned.

I understand that this is how vBulletin works and it's not vBSEO's fault, but could it be solved somehow?

PageUp · #2 11-02-2005, 04:59 PM

I may have found a solution. Using this script in showthread and forumdisplay pages, all rewrited URLs with query string will not be indexed and thus preventing double content issues.

For instance, an URL like forum/forumdisplay.php?f=8&page=3&sort=lastpost&order=&p p=20&daysprune=-1 rewrites to something like forum/forum8/index3.html?sort=username&order=&pp=20&daysprune=-1. Let's say that this sorting is not the same as our current default sorting. This just confuses the spider and can lead to double content penalty, I think, escpecially if you changed the default sorting settings. What this script does is insert nofollow,noindex tags into it so it's not indexed. It doesn't add these tags in "normal" pages, like forum/forum8/index3.html.

PHP Code:

  ob_start();
//print_r($_GET);
$matchingArr["t"]="set";
$matchingArr["page"]="set";
$matchingArr["threadid"]="set";
$matchingArr["pagenumber"]="set";
$matchingArr["f"]="set";
$matchingArr["forumid"]="set";
////////////////////////////////
$bDisplayMeta = 0;
foreach ($_GET as $key1 => $myKey) 
{
   $bOk = 0;  
   foreach ($matchingArr as $key2 => $myKey2) 
 if ($key1==$key2)
    $bOk=1;
   if ($bOk==0)
      $bDisplayMeta = 1;   
}
if($bDisplayMeta) {
echo "<META NAME=\"ROBOTS\" CONTENT=\"NOINDEX, NOFOLLOW\" />";
}
$meta_robots = ob_get_contents();
ob_end_clean();

What do you guys think?

Oleg Ignatiuk · #3 11-03-2005, 10:00 PM

Hello PageUp!

Thank you for this suggestion.

A new feature has been added to vBSEO to avoid duplicate content (will be available in the next release) that will make a 301-redirection to a *clean* URL when SE bot is requesting a link with custom page size parameter ("pp").
Normal users will still be able to browse forums with different page sizes.

When the thread page higher than a total count of pages is requested, it is *redirected* to the last page now.

PageUp · #4 11-04-2005, 04:08 AM

Sounds perfect! Thanks for taking my concerns seriously.

But what about if the SE bot requests a forumdisplay page with custom daysprune, sort and order settings (e.g. forumdisplay.php?f=8&page=3&sort=lastpost&order=as c&p p=20&daysprune=-1)? Will that redirect to a clean URL too?

Oleg Ignatiuk · #5 11-04-2005, 01:38 PM

The same is applied to "sort", "order", "daysprune" parameters.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Possible duplicate content issue?	MSJA	General Discussion	5	07-01-2006 09:06 AM
Matt Cutts says Honest site owners don't really have to worry about duplicate content	BamaStangGuy	General Discussion	24	06-05-2006 03:01 PM
2006 SES Conference - Duplicate Content Session	msimonds	Member Articles	0	03-07-2006 05:04 PM
Question regarding linking to styles ? duplicate content	s2kinteg916	General Discussion	0	11-11-2005 12:41 AM

SEO

Possible duplicate content

Possible duplicate content

Similar Threads