Google Sitemap-Listing of old url's option-from 1.7

When upgrading your vBSEO Google/Yahoo Sitemap Generator from version 1.6.1 to 1.7.0, there is a new option to include both the rewritten vBSEO urls and the old urls. This is a new feature I recommended to the vBSEO team, so here is an analysis of why I feel you should use this feature:

Why list the old urls?

Simply:

Google needs to be able to see the 301 redirect from the old to the new url, and so delete the old url from its index.

If the old urls on the forum have been totally rewritten by vBSEO, then Google is not able to find the old urls and will not respider the old urls for awhile. The old urls will exist and will, therefore, be duplicate content to the new urls until the old urls are respidered and the redirects found.

It will, however, find the old urls that have external links into them.

Therefore:

Best to have a vBSEO sitemap that includes each of the old urls and so have a way of telling Google to spider the old urls and find the 301 redirects. It will, therefore, delete the old urls from its index. The deletion won't be immediate, as Google treats the deletion as a separate batch process.

When to stop including the old urls:

Stop including the old urls once Google stops showing them on your site:domain.com search.The vBSEO Google/Yahoo Sitemap will show the main old urls, but not all the possibilities, so there will still be some that Google needs to look after itself with its own respidering process.

Complications

Even though you may have a sitemap, those urls may not be spidered that fast, and not all the urls may be spidered. You need to make sure that you have good Google PR (probably PR5+) on your index page plus lots of external links to internal pages. This will get all your urls spidered well.

See the following on Matt Cutts blog 27 Mar 2006:

Q&A Thread

Q: “My sitemap has about 1350 urls in it. . . . . its been around for 2+ years, but I cannot seem to get all the pages indexed. Am I missing something here?”

A: One of the classic crawling strategies that Google has used is the amount of PageRank on your pages. So just because your site has been around for a couple years (or that you submit a sitemap), that doesn’t mean that we’ll automatically crawl every page on your site. In general, getting good quality links would probably help us know to crawl your site more deeply. You might also want to look at the remaining unindexed urls; do they have a ton of parameters (we typically prefer urls with 1-2 parameters)? Is there a robots.txt? Is it possible to reach the unindexed urls easily by following static text links (no Flash, JavaScript, AJAX, cookies, frames, etc. in the way)? That’s what I would recommend looking at.

All the best.