Robots.txt now support the sitemap file

Hello,
I just see this in the google blog and it`s really useful since we won`t have to submit our sitemaps to every search engine individual anymore

it's now easier for you to tell us where your Sitemaps live. We wondered if we could make it so easy that you wouldn't even have to tell us and every other search engine that supports Sitemaps. But how? Well, every website can have a robots.txt file in a standard location, so we decided to let you tell us about your Sitemap in the robots.txt file. All you have to do is add a line like

Sitemap: http://www.mysite.com/sitemap.xml

to your robots.txt file. Just make sure you include the full URL, including the http://. That's it. Of course, we still think it's useful to submit your Sitemap through Webmaster tools so you can make sure that the Sitemap was processed without any issues and you can get additional statistics about your site

Official Google Webmaster Central Blog: What's new with Sitemaps.org?

Very cool. They've really simplied the process, and I'm sure other search engines will now follow.

Looks good, and also Ask.com and Microsoft Live are now supporting sitemaps through autodiscovery (as well as yahoo/google). Which means all the major search engines will know about every page of your site soon!

Awesome

vBSEO Sitemap Generator's sitemap index file is: sitemap_index.xml.gz

.gz format supported?

Yes, Google supports gzip compressed sitemaps.

Originally Posted by Oleg Ignatiuk

Yes, Google supports gzip compressed sitemaps.

I was refferring to this:

"We wondered if we could make it so easy that you wouldn't even have to tell us and every other search engine that supports Sitemaps."

but did a research and gzip seems to be supported by the other search engines, too.

Added my line as Sitemap: http://www.mysite.com/sitemap.xml.gz

Thanks for posting this information.

Hi,

Please excuse me because i am newbie.

2) i have never make a robots.txt for my vbulletin, how can i do this one and what code can i put in it ?
1) where can i put the robots.txt ? in the root site or in forum folder ?

thanks

file should go in root, and it is important to have one because virtually all spiders will first search to see if this file exists on your site to see if there are specific instructions for them to follow.

What you can or should put in there is a debate that will go on for years, and partly depends on what you have in your site that you don't want search engines indexing.

Here is an example from one of my sites

Code:

User-agent: *
Disallow: /forum/vbseo_sitemap
Disallow: /forum/admincp
Disallow: /forum/attachments
Disallow: /forum/attachment.php
Disallow: /forum/arcade.php
Disallow: /forum/calendar.php?do=add
Disallow: /forum/cron.php
Disallow: /forum/editpost.php
Disallow: /forum/login.php
Disallow: /forum/modcp
Disallow: /forum/moderator.php
Disallow: /forum/membermap.php
Disallow: /forum/newreply.php
Disallow: /forum/newthread.php
Disallow: /forum/online.php
Disallow: /forum/payments.php
Disallow: /forum/pda
Disallow: /forum/postings.php
Disallow: /forum/printthread.php
Disallow: /forum/private.php
Disallow: /forum/profile.php
Disallow: /forum/register.php
Disallow: /forum/report.php
Disallow: /forum/reputation.php
Disallow: /forum/sendtofriend.php
Disallow: /forum/search.php
Disallow: /forum/sendmessage.php
Disallow: /forum/showpost.php
Disallow: /forum/subscription.php
Disallow: /forum/threadrate.php
Disallow: /forum/usercp.php
Disallow: /forum/spy.php
Disallow: /forum/tags/

Here is an example of someone just blocking specific spiders

Code:

User-agent: BoardTracker
Disallow: /

User-agent: Gigabot
Disallow: /

Or, you could simply have a robots.txt file that is blank. You don't have to have anything in it, but most believe it is helpful to restrict spiders from going through unimportant files, and/or files that don't need to be seen by the public.

Having a robots.txt file that is blank simply tells spiders that everything is available to be searched.

You should list your sitemap index file not your individual sitemaps.

Thanks

No I have not a private forum ...

so i have make a robots.txt with this only code :

Code:

Sitemap: http://www.mysite.com/forum/vbseo_sitemap/data/sitemap_index.xml.gz

and put it in my root :
www.mysite.com/robots.txt

So all is ok ?

excuse this duplicate post...

Originally Posted by rinka128

Thanks

No I have not a private forum ...

so i have make a robots.txt with this only code :

Code:

Sitemap: http://www.mysite.com/forum/vbseo_sitemap/data/sitemap_index.xml.gz

and put it in my root :
www.mysite.com/robots.txt

So all is ok ?

No, you should use:

Sitemap: http://www.mysite.com/forum/sitemap_index.xml.gz

Originally Posted by rinka128

Thanks

No I have not a private forum ...

so i have make a robots.txt with this only code :

Code:

Sitemap: http://www.mysite.com/forum/vbseo_sitemap/data/sitemap_index.xml.gz

and put it in my root :
www.mysite.com/robots.txt

So all is ok ?

no, you only want to link to to the root, as vbseo has the .htaccess condition to re-write it to its 'real' location.

http://www.mysite.com/forum/sitemap_index.xml.gz
or
http://www.mysite.com/sitemap_index.xml.gz

Ok ! thanks :-)