Am I doing something wrong?

According to my sitemap spider logs, Google seems to only crawl 50-100 (if I am lucky) pages per day.

Yahoo does several thousand.

Is there maybe something I am doing wrong to cause G to not show me the love? Main page PR5, a few PR4 main pages...that should mean they come more often than they are.

Hello,

just in case, you can check the googlebot crawl stats for your site in Google Webmaster Account: http://www.google.com/webmasters/sitemaps/siteoverview.

At least you will see "official" crawl stats and the crawl errors:
HTTP errors
Unreachable URLs
URLs restricted by robots.txt
URLs not followed
URLs timed out

Hmm... errors:

Originally Posted by Google Sitemaps

Below are pages that we tried to crawl (found either through links from your Sitemaps or from other pages) but couldn't access.
http://www.nzboards.com/forums/membe...rklechick.html Sitemap Web General HTTP error Jan 4
http://www.nzboards.com/forums/members/amystery.html Sitemap Web General HTTP error Jan 3
http://www.nzboards.com/forums/members/covosat.html Sitemap Web General HTTP error Jan 3
http://www.nzboards.com/forums/members/fatcash.html Web General HTTP error Jan 3
http://www.nzboards.com/forums/members/latina17.html Web General HTTP error Jan 3
http://www.nzboards.com/forums/members/nitefenix.html Sitemap Web General HTTP error Jan 3
http://www.nzboards.com/forums/21010-post3.html Web 404 not found Jan 1

Where is it getting those URLs from? :(

Perhaps these URLs were catched by googlebot while you have tried different URL formats in vbseocp? Anyway, having a few errors like these is not a big problem, I believe.
Or do you have a lot of them?

There are fewer errors than a page full .. and yes, they do appear to be related to changes in URL format I have made.

I'm feeling a lot more secure now. Cheers Oleg.

Gotta now wonder why Google doesn't crawl it as much as I would like.

Dammit. Those URLs are still showing up in my sitemap errors page, and all between Jan 15 and today. :(

Heres my sitemap_index.xml:

Code:

<?xml version='1.0' encoding='UTF-8'?>
<sitemapindex xmlns="http://www.google.com/schemas/sitemap/0.84"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.google.com/schemas/sitemap/0.84
http://http://www.google.com/schemas/sitemap/0.84/siteindex.xsd">
<sitemap>http://www.nzboards.com/sitemap_1.xml.gz</sitemap>

</sitemapindex>

Notice the http://http://www.google.com part? Will that make a difference?

Plus, here is something very scary from sitemap_2.xml...

<loc>http://www.nzboards.com/forums/members/amystery.html</loc>

. That is NOT following the rewrite rule I have for members. It is

Code:

members/[user_name]/

.

Notice the http://http://www.google.com part? Will that make a difference?

That's fine - this is a part of xml structure definition.

Plus, here is something very scary from sitemap_2.xml...

sitemap_2? But in sitemap index you have only one sitemap file - sitemap_1.xml.gz! Perhaps you have sitemap_2.xml.gz from one of the previously generated sitemaps with smalled number of pages per file? You should remove it if that is the case.

Nice spotting! I have removed it. We will see what happens now.