Should G be spidering the archives or should I block it via robots.txt? What's the current thinking on this as it relates to duplicate content?
vBulletin 4, the most powerful community software + vBSEO 3.5, the ultimate SEO solution = Your ultimate platform for 2010 and beyond. Click below to learn more.
This is a discussion on the archives within the General Discussion forums, part of the vBulletin SEO Discussion category; Should G be spidering the archives or should I block it via robots.txt? What's the current thinking on this as ...
Should G be spidering the archives or should I block it via robots.txt? What's the current thinking on this as it relates to duplicate content?
I recommend:
turn off in vb ACP
don't re-write in vbseo
exclude in robots.txt (if you already have them indexed)