Should /forums/tags be blocked in robots.txt?

[bfseo]

Just wonder if any of you block /forums/tags in your robotx.txt file.

Is there any reason why one would want to make this available for indexing?

[Brian Cummiskey]

I don't block them, but i don't include tag pages in sitemaps.

[bfseo]

Hmm...my sitemap page includes them. I'm referring to http://www.mydomain.com/forums/sitemap_index.xml.gz

How do I submit a site map to Google without the tag pages? Submit three separately for (forum, cms, and blog)?

[Marco Mamdouh]

You just need to disable it from

vBulletin Admincp > vBSEO sitemap generator settings > Include Tags > Set it to "NO"
Then go to your ftp and remove all .gz files located in this folder vbseo_sitemap/data/
And then re-generate your sitemap again, It will not include tags pages any more.

[bfseo]

This might sound like a stupid question.

After I generate the new sitemap, will it be picked up automatically?

Or do I have to in addition go to Google/Yahoo and submit sitemaps there as well?

[Brian Cummiskey]

If you've already submitted your sitemap in webmaster tools, you don't need to do it again.

When run, it pings the services to tell them that new files exist for download.

[bfseo]

Will results be duplicated in search engines if its submitted in webmaster tools and the sitemap is generated in the vb admin cp as well?

[Brian Cummiskey]

I don't know how that at all correlates....

No.

The sitemap tool RUNS and CREATES the sitemap.
webmasters tools is for ANALYZING the sitemap, and other aspects of the site.

[bfseo]

Brian, it's been about a month since vBSEO was installed and my site still looks exactly the same in search engines. May I PM you my website address so you can check it out in search engines and give me some advice regarding what I may be doing wrong? I don't believe my settings are set up the way they should be.

[Brian Cummiskey]

As a staff member, I can see your license urls. It can take 4-8 weeks in most cases to start seeing changes. I DO see a full page of seo-formatted urls in google using the site: command for your domain. It's working.. it just takes time.

[bfseo]

Thanks.

When I run a search under my domain name, I get some ugly results.

There's....

MyDomain Name (This is fine)
Description of CMS

Forum - My Domain Name Community
Description of first forum on main forum page (I don't want this)

Name of Content - My Domain Name (Why is it doing this with an article?)
- sublink to thread
- sublink to thread
- sublink to thread

Smilies page (Why is it still showing up if I've put in a command in robots.txt not to crawl this page?)

It's frustrating. I've added several items to robots.txt but it's still picking them up in search engines.

And many of my CMS articles are showing up as "Article - Name of Article."

If there's something I can do now to start cleaning these things up, I'd like to start doing it.

I'm starting to feel like I should can the VB CMS.

[Brian Cummiskey]

changes in the search engine, webmaster tools, and robots.txt are not instant. All these things can take 2 months to change or go into effect. You need to be patient and let it run its course. Google has 23423823823923455378634578634785 other websites too look at too.

[mikrac]

don't block them

[Marvin Hlavac]

I don't block them, but i don't include tag pages in sitemaps.

Hi Brian,

Two questions, if I may:

1. I also think there's no need to include TAG pages in sitemaps. Search engines will find them anyway via the TAG links on the bottom of thread pages. But I'd like to know your opinion, your reasoning for not including them. Does including / not including of pages in sitemap signifies the importance (or the lack of it) of the pages to search engines?

2. But more importantly, I'd like your current opinion on if we should have TAG pages at all? And if yes, would it make any sense to use the NOINDEX meta tag on them?

It seems some people have recently been removing pages from their sites they feel may be considered as "thin content" by G**gle (pages such as TAG, or category pages). Alternativelly, some people have been suggesting to use the "noindex" meta tag on these pages.

I've decided against making a quick decision on this, but rather to see what more people have to say on the subject.

[Brian Cummiskey]

Tag pages are effectively 'search results' pages. They aren't exactly 'content' pages, and thus i don't think they belong in the sitemap.

I don't think it's necessary to block them or hide them in anyway, but i don't think they should be lumped in as a content page.