robots.txt

i've just created a new robots.txt (that i'd put onto the main folder - /public_html) this is the code:

Code:

User-agent: *
Allow: /
Sitemap: http://www.fniv.it/board/sitemap_index.xml.gz

it's correct? any suggestion? (i've grabbed the sitemap's URL from the ACP)

but i saw one thing in the GTools: when i run a test against my board domain (FNIV - Federazione Italiana Videogiocatori) i got this result:

Allowed by line 2: Allow: /
Detected as a directory; specific files may have different restrictions

so if i change the sitemap's URL in the robots.txt and put:

Code:

Sitemap: http://www.fniv.it/sitemap_index.xml.gz

and run the test again, now i got this result:

Allowed by line 2: Allow: /
Detected as a directory; specific files may have different restrictions

and (Parse Results)

Line 3: Sitemap: http://www.fniv.it/sitemap_index.xml.gz Valid Sitemap reference detected

then i'm wondering about this, what's the correct robots.txt sitemap's URL?

Hello KURTZ,


It seems there is a problem with the "/"

Could you try?

Sitemap: http://www.fniv.it/board/sitemap_index.xml.gz

with no Allow: /

Also, trying to submit http://www.fniv.it/board/xmlsitemap.php via GTools would be good since it's easier and really effective. (Tried it myself!)

Does that work?

Sitemap: /board/sitemap_index.xml.gz

result:

Allowed by line 2: Allow: /
Detected as a directory; specific files may have different restrictions

and

Invalid sitemap URL detected; syntax not understood

Sitemap: http://www.fniv.it/board/sitemap/

Allowed by line 2: Allow: /
Detected as a directory; specific files may have different restrictions

(check the pic, i have a different URL)

same thing with this:

http://www.fniv.it/board/xmlsitemap.php

Sorry I edited my previous post a bit late.

It seems there is a problem with the "/"

Could you try?
Sitemap: http://www.fniv.it/board/sitemap_index.xml.gz

Deleting Allow: /

Code:

User-agent: *
Sitemap: http://www.fniv.it/board/sitemap_index.xml.gz

There was really no problem at first. That's just google's way of saying the directory is available but other specific directories/files can have different restrictions (allows or disallows).

see also:
Tutorial how to make a good robots.txt file

Originally Posted by Peter_Rosado

Sorry I edited my previous post a bit late.



Deleting Allow: /

Code:

User-agent: *
Sitemap: http://www.fniv.it/board/sitemap_index.xml.gz

There was really no problem at first. That's just google's way of saying the directory is available but other specific directories/files can have different restrictions (allows or disallows).

just tried this:

Code:

User-agent: *
Sitemap: http://www.fniv.it/board/sitemap_index.xml.gz

result:

Allowed
Detected as a directory; specific files may have different restrictions

Originally Posted by Brian Cummiskey

see also:
Tutorial how to make a good robots.txt file

thanks Brian just tested this:

Code:

# Allow Archive.org to save snapshots of everything
User-agent: ia_archiver
Allow: /

# Tame yahoo... it tends to eat a ton of resources without a delay
User-agent: Slurp
Crawl-delay: 60


#list individual pages and files here that all bots should ignore, as well as group extentions.
#If you re-write everything to .html, you can disallow *.php, but note that if you don't have a CRR for custom pages, those will be blocked.

User-agent: *
Disallow: *.js
Disallow: /board/clientscript/
Disallow: /board/cpstyles/
Disallow: /board/customavatars/
Disallow: /board/customprofilepics/
Disallow: /board/images/
Disallow: /board/ajax.php
Disallow: /board/attachment.php
Disallow: /board/calendar.php
Disallow: /board/cron.php
Disallow: /board/editpost.php
Disallow: /board/global.php
Disallow: /board/image.php
Disallow: /board/inlinemod.php
Disallow: /board/joinrequests.php
Disallow: /board/login.php
Disallow: /board/member.php
Disallow: /board/memberlist.php
Disallow: /board/misc.php
Disallow: /board/moderator.php
Disallow: /board/newattachment.php
Disallow: /board/newreply.php
Disallow: /board/newthread.php
Disallow: /board/online.php
Disallow: /board/poll.php
Disallow: /board/postings.php
Disallow: /board/printthread.php
Disallow: /board/private.php
Disallow: /board/profile.php
Disallow: /board/register.php
Disallow: /board/report.php
Disallow: /board/reputation.php
Disallow: /board/search.php
Disallow: /board/sendmessage.php
Disallow: /board/showgroups.php
Disallow: /board/subscription.php
Disallow: /board/threadrate.php
Disallow: /board/usercp.php
Disallow: /board/usernote.php

#Finally, list the path to your sitemap:
Sitemap: http://www.fniv.it/board/sitemap_index.xml.gz

and got this:

Allowed
Detected as a directory; specific files may have different restrictions

and this for parse:

Line 7: Crawl-delay: 60 Rule ignored by Googlebot

it's fine?

latest quesion: what about this rule?

Code:

Disallow: *.js

i should put /board/ before the *.js?

1) The warning is ok.
2) this targets slurp, aka yahoo. google will ignore it
3) this keeps your javascript files out of the index. You can use root level (will block all javascript sitewide from being indexed)

Originally Posted by Brian Cummiskey

1) The warning is ok.
2) this targets slurp, aka yahoo. google will ignore it
3) this keeps your javascript files out of the index. You can use root level (will block all javascript sitewide from being indexed)

well done Brian!