Should G be spidering the archives or should I block it via robots.txt? What's the current thinking on this as it relates to duplicate content?
This is a discussion on the archives within the General Discussion forums, part of the vBulletin SEO Discussion category; Should G be spidering the archives or should I block it via robots.txt? What's the current thinking on this as ...
Should G be spidering the archives or should I block it via robots.txt? What's the current thinking on this as it relates to duplicate content?
I recommend:
turn off in vb ACP
don't re-write in vbseo
exclude in robots.txt (if you already have them indexed)