Calendar

I noticed that google has indexed a million blank pages of our calender from our brand new site. The calendar contains nothing and not sure it ever will much. Advise...

thanks!

You can add calendar.php to your robots.txt file and prevent SE's indexing them.

I have a normal website with a robots.txt file in the root of that (saying to follow everything) and then I have the "forum" folder with the vB. Do I update the robots.txt in the root of the website file or one somewhere else?

The one in the root of my website contains this:

Code:

User-agent: *
Disallow:
Sitemap: http://mywebsite/sitemap.xml

So if that IS the one I update then what should it look like after the edits exactly?

It appears I also have some private forums getting indexed although when I setup the forums permissions in vB I checked for it not to do the search option? Not to mention the forums are hidden except to specific user groups. Not to mention my "mod" forums. How can I be possitive they don't get indexed anymore?

Thanks!

robots.txt only functions in root. You will need to path accordingly FROM root.

It is useless to allow everything... that is default behavior.
Tutorial how to make a good robots.txt file - vBulletin SEO Forums

It is impossible for private forums (not accesible to human guests) to be indexed unless you installed some bot as member cloaking mod.

Originally Posted by Mert Gökçeimam

You can add calendar.php to your robots.txt file and prevent SE's indexing them.

I never use my calendar for anything either so I think I will turn that off as well in my robots.txt file.

So would the correct way to do say just be:

User-agent: *
Disallow: *.js
Disallow: search.php
Disallow: /includes/
Disallow: /install/
Disallow: /customavatars/
Disallow: /vbcms-comments/
Disallow: /calendar.php

Or do I need a slash / at the end like some of the others?

/ is only used for folders (physcial, or logical), not individual scripts. You are fine as above.

Ok got it thanks Brian

I was facing a similar problem. Now I understand that I have to turn off calendar in order to prevent google from indexing it.

Because I have my normal non-cms website with the forum attached then I have 2 sitemaps. 1 for my normal site and then one for the forum that vBSEO creates. Am I doing something wrong having 2? Which one do I point to in my robots.txt file?

Thanks!

You can put both in.

ie,
Sitemap: http://mywebsite/sitemap.xml
Sitemap: http://mywebsite/forum/sitemap_index.xml.gz

ok, thanks. And if I left all the paths/names default then sitemap_index.xml.gz is the correct extension and everything correct?

***wait***

just checked and the sitemap seems to be in store_sitemap? So would it be:

http://mywebsite/forums/store_sitema..._index.xml.gz?

and what are:

vbulletin_sitemap_forum_1.xml.gz

and

vbulletin_sitemap_thread_0.xml.gz


Do they need to be referenced in the robots.txt as well?

Thanks!

Those are vb sitemaps and should be disabled and those files deleted when using vbseo. The vbseo sitemap replaces vb's built in tool.


Put the url for your store to whatever the actual url to load it is.

I had just came across that you should make sure your vB default sitemap is off a few mins ago and so I turned it off. But I didn't realize those were the files it generated. I'll delete them. Delete ALL of them including the sitemap_index.xml.gz?

Originally Posted by Brian Cummiskey

Put the url for your store to whatever the actual url to load it is.

Not sure what you mean. I don't have a "store". It's just a vB forum only?

vb writes it's sitemap files to ROOT. vbseo sitemap writes to vbseo_sitemap/data/. Do not delete the vbseo files.

I have no idea what you're tlaking about with the store either. you brought it up.

just checked and the sitemap seems to be in store_sitemap? So would it be:

http://mywebsite/forums/store_sitema..._index.xml.gz?

if you don't have a store, why do you have a store_sitemap?

I have no idea what you're doing.


Put the vbseo sitemap index in the robots file.

If you have another one, list that path too, whatever or where ever it may physically be on your site.

Sorry, I don't know what I'm doing either.

I looked it up and store_sitemap is just the folder vB created by default for the sitemap files. It's in my admin and had nothing to do with me at all... I'll delete the folder.

So if I view my vBSEO stats at the top right corner of the screen it states that the sitemap is here:

http://www.mysite/forums/sitemap_index.xml.gz

But if I click on that link it does nothing. If I go to that address I see nothing. If I look in my hosting there is nothing there.


If I follow your path in my hosting account then there is a file here:

http://www.mysite.com/forums/vbseo_s...p_index.xml.gz

But it says it's only 343 bytes and if I open it in transmit or text edit it appears to be blank. If I go to that url in my browser then it just brings up "Forbidden".

So is that right?

Based off this, what is the exact URL for the vBSEO that I should have in my robots.txt?