Another Robots.txt question...

In my robots.txt file... when ran through an optimizer it came back as follows:

Line 4 User-agent: Slurp
You specified both the generic user-agent "*" and specific user-agents for this block of code; this could be misinterpreted.
Line 5 Crawl-delay: 60
Line 6
Line 7 # Disallow directory
Line 8 User-agent: *
You specified both the generic user-agent "*" and specific user-agents for this block of code; this could be misinterpreted.

Does this mean I need to take either User-agent: Slurp or User-agent: * out and not have both?

Does it really hurt to have both? I only ask because I saw it blessed by vBSEO staff here in this forum on more than one occasion.

Thanks!

Hey Sonnie,

What is actually the full contents of your robots.txt, the messages makes little sense to me without something to reference against.

Ceri

Thanks Ceri... here is our full robots.txt file...

# For domain: Home Theater Forum - Home Theater Systems - HomeTheaterShack
# All robots will spider the domain

User-agent: Slurp

Crawl-delay: 60

# Disallow directory

User-agent: *

Disallow: /attachment.php
Disallow: /attachments/
Disallow: /cgi-bin/
Disallow: /deniedaccess.html
Disallow: /display.html
Disallow: /forums/admincp/
Disallow: /forums/ajax.php
Disallow: /forums/articlebot/
Disallow: /forums/clientscript/
Disallow: /forums/cpstyles/
Disallow: /forums/customavatars/
Disallow: /forums/customprofilepics/
Disallow: /forums/calendar.php
Disallow: /forums/cron.php
Disallow: /forums/editpost.php
Disallow: /forums/global.inc.php
Disallow: /forums/includes/
Disallow: /forums/inlinemod.php
Disallow: /forums/install/
Disallow: /forums/ipinfo.php
Disallow: /forums/joinrequests.php
Disallow: /forums/login.php
Disallow: /forums/markers.xml
Disallow: /forums/member.php
Disallow: /forums/memberlist.php
Disallow: /forums/misc.php
Disallow: /forums/modcp/
Disallow: /forums/moderator.php
Disallow: /forums/moderatorapplication.php
Disallow: /forums/modules/
Disallow: /forums/newattachment.php
Disallow: /forums/newreply.php
Disallow: /forums/newthread.php
Disallow: /forums/online.php
Disallow: /forums/payment_gateway.php
Disallow: /forums/payments.php
Disallow: /forums/poll.php
Disallow: /forums/postings.php
Disallow: /forums/private.php
Disallow: /forums/process.php
Disallow: /forums/profile.php
Disallow: /forums/register.php
Disallow: /forums/report.php
Disallow: /forums/reputation.php
Disallow: /forums/search.php
Disallow: /forums/sendmessage.php
Disallow: /forums/showgroups.php
Disallow: /forums/subscription.php
Disallow: /forums/test.php
Disallow: /forums/threadrate.php
Disallow: /forums/usercp.php
Disallow: /forums/usernote.php
Disallow: /forums/vbgooglemapme.php
Disallow: /forums/vbseocp.php
Disallow: /ModeratorOptions.pdf
Disallow: /newrsanalog.cal
Disallow: /newrsdigital.cal
Disallow: /oldrsanalog.cal
Disallow: /oldindex.html
Disallow: /openads/
Disallow: /phpform/
Disallow: /roomeq/oldindex.html
Disallow: /ssm.js
Disallow: /ssmItems.js
Disallow: /subscriptions/
Disallow: /test.html

There is nothing wrong with that file, with the exception that you reference your modcp and admincp directories, which can be a secruity issue. But on the robots side of things, it's completely valid.

With your robots.txt Slurp will be able to spider all files and directories blocked for other robots.
Either copy all ' User-agent: *' rules for slurp as well or just remove 'User-agent: Slurp' and leave crawl-delay.
that directive is ignored by googlebot.

Originally Posted by Brian Cummiskey

There is nothing wrong with that file, with the exception that you reference your modcp and admincp directories, which can be a secruity issue. But on the robots side of things, it's completely valid.

So I should remove admincp and modcp? Again... I copied it from here and it was previously given blessings from here, but it could have easily been overlooked.

Originally Posted by Notorious

With your robots.txt Slurp will be able to spider all files and directories blocked for other robots.
Either copy all ' User-agent: *' rules for slurp as well or just remove 'User-agent: Slurp' and leave crawl-delay.
that directive is ignored by googlebot.

Thanks!

Yeah, you should never include private directories. You just told would-be hackers where your panel is.

I noticed you had them in yours when I searched for a few examples.

are you sure?

You are right... I was looking at one from briansol. Sorry about that.

What is ia_archiver?

And have you ever heard of sojourner?

ia_archiver = archive.org bot.

no idea what sojourner is.

What is the purpose of ia_archiver... why add that?

sojourner constantly crawls my forum.

archive.org takes snapshots of your site. it's neat to look back on where you've been.
CRAWLABILITY.com
vBulletin Search Engine Optimization Forums

They tend to not cache a lot of images or linked css/etc.