Robots.txt now support the sitemap file

Originally Posted by jbartle

I know this is a bit late, but wouldn't it be better to write it as:
Sitemap: /forums/sitemap_index.xml.gz

instead of putting the whole url?

No, read the part of the article quoted in the first post of the thread. You need to put the entire address in, including the http://

Robots.txt now support the sitemap file

Yeah I read it...which is why I asked if it would be better...thanks.

Interesting.

I looked in the folder and saw in the data folder with the zipped sitemaps a .htaccess file that has written inside it

deny from all

.

What's that for and does it influence anything?

Also, I see 2 more zipped files

one is called urllist.txt.gz and the other is sitemap_1.xml.gz

so what are they about and do they need to go in the robots.txt file to?

What's that for and does it influence anything?

It's required to disallow direct access to the data/ folder.

urllist.txt.gz is for Yahoo submission (now Yahoo supports xml sitemaps too though)
sitemap_1.xml.gz is an XML sitemap
sitemap_index.xml.gz is an index (list) of all xml sitemap files generated

So should we put this one ( sitemap_1.xml.gz) in the robots.txt file as well?

All sitemap_X.xml.gz files should be discovered automatically from the index file.