Google indexed pages going down

My Google indexed links are going down. They have dropped about 1K in the last few days and I haven't done anything different. I do use the www as my primary domain and it seems Google has indexed both URL's although I specified a primary domain and only submit sitemaps through the www, so this could have something to do with it, but the www has gone down too. Does anyone know what else it could be?

You should set up your site to only have 1 url variant, in addition to the webmasters tool setting.
[How to] Setup www.yourdomain.com or yourdomain.com (www vs. non-www) - With .htaccess 301 redirects - vBulletin SEO Forums

You are a fairly new install, so this is common to see a loss of urls (std vb has tons of duplicates).

The metric to watch is traffic, and specifically google/search referrers.