Sitemap / Robots / Tags questions

I have these lines in my robots.txt:

Sitemap:xttp://www.vipmoneymakersclub.com/sitemap_index.xml.gz
Sitemap:VIP Money Makers Club - Where Money Makers Are Born!

are these files supposed to exist? I thought my sitemap was working
properly, but now I am not so sure, if not then why would a url
be pointing to sitemap files that do not exist? By looking at the
vbseo sitemap settings, it does to be working correctly. I am just
curious on why the sitemap files appear not to exist....

Tags:

I also notice urls and tags that contain % in the url
is this format normal? Why are there % signs in some
urls and not there for other urls.

Robots:

My robots file has:

disallow: /tags/

Why would anyone want to disallow their tags? Is this not why
we have tags so they can be searched and indexed by engines?

Thanks

ps. does anyone have an updated robots.txt? I have been running across
so many robot specs lately, I am not even sure what a good one is if I
actually seen the perfect one; I really would not even know.

Originally Posted by gotlinks

I have these lines in my robots.txt:

Sitemap:xttp://www.vipmoneymakersclub.com/sitemap_index.xml.gz
Sitemap:VIP Money Makers Club - Where Money Makers Are Born!

first line is fine.

2nd parsed, so i have no idea what it was, but its probably not supposed to be there.

Tags:

I also notice urls and tags that contain % in the url
is this format normal? Why are there % signs in some
urls and not there for other urls.

% is used as a hex code in urls. ie, %20 is a space.

Robots:

My robots file has:

disallow: /tags/

Why would anyone want to disallow their tags? Is this not why
we have tags so they can be searched and indexed by engines?

Thanks

since no robots file ships with vb/vbseo, you must've added that at some point on your own.

There's no such thing as a generic robots file as everyones site is different and some folks want to block other things that others want open. There's a basic format tutorial here:

Tutorial how to make a good robots.txt file - vBulletin SEO Forums

ok...

so is the %20 supposed to be there? I mainly find them in tags, I do do think
I have seen them in main urls, but they look strange, how to fix? if i need too...

I also noticed that I have disallow on member.php and /members/ and I
clearly notice that spiders are looking at members, how come?

Thanks

if you tag a thread as "some tag" it will url convert to "some%20tag" by default.

You can enable the tag filter in vbseocp.php#char_opt , advanced tab, but it may have varied results.


Robots.txt is merely a 'suggestion'.

If you're feeding member urls via the sitemap, it's going to at least try to look at it. Make sure all your items say the same thing permissions-wise.