Google acting wierd on my robots.txt?

Hi all,

I have this robots.txt file:

Code:

User-agent: *
Disallow: /apc.php
Disallow: /forum/admincp/
Disallow: /forum/cgi-bin/
Disallow: /forum/clientscript/
Disallow: /forum/cpstyles/
Disallow: /forum/customavatars/
Disallow: /forum/customprofilepics/
Disallow: /forum/images/
Disallow: /forum/includes/
Disallow: /forum/modcp/
Disallow: /forum/ajax.php
Disallow: /forum/attachment.php
Disallow: /forum/cron.php
Disallow: /forum/editpost.php
Disallow: /forum/faq.php
Disallow: /forum/global.php
Disallow: /forum/infraction.php
Disallow: /forum/inlinemod.php
Disallow: /forum/joinrequests.php
Disallow: /forum/login.php
Disallow: /forum/misc.php
Disallow: /forum/moderation.php
Disallow: /forum/moderator.php
Disallow: /forum/newattachment.php
Disallow: /forum/newreply.php
Disallow: /forum/newthread.php
Disallow: /forum/payment_gateway.php
Disallow: /forum/payments.php
Disallow: /forum/poll.php
Disallow: /forum/postings.php
Disallow: /forum/private.php
Disallow: /forum/register.php
Disallow: /forum/report.php
Disallow: /forum/reputation.php
Disallow: /forum/search.php
Disallow: /forum/sendmessage.php
Disallow: /forum/showgroups.php
Disallow: /forum/subscription.php
Disallow: /forum/threadrate.php
Disallow: /forum/usercp.php
Disallow: /forum/usernote.php

Why Google indexed the apc.php file? And why it did not indexed the main forum page?
site:adolympics.com - Google Search

Thanks for explaining. Also, let me know if I should add/remove lines to the robots file.

Hello Floren,

it is possible that Google indexed your apc.php link before you added it to robots.txt and just haven't updated the index yet.

Thanks for the reply, Oleg.
I don't think it was indexed before I added the robots.txt file. From what I see, the apc.php was indexed on July 7th.
My robots.txt file is there since March?

Do you have any idea why it did not catched the main forum page?

Google will sometimes index pages based on the fact that there are a lot of inbound links pointing to it. Matt Cutts actually talked about this phenomina briefly at threadwatch.org not too long ago. He brought up the way things were several years ago with ebay and places like the CA DMV not allowing robots in. They couldn't violate robots.txt, so they ended up just making assumptions about pages based on external info. Since then, google's behavior in that respect has not changed much. Notice that the page is not actually cached. I doubt searching through your logs if you'll see that the googlebot got there.

The links to your APC.php file came from a discussion you apparently had at simplemachines.org and another one at dreamhost.com around June 12th. Kind of creepy that I know about your message board interactions isn't it?

As far as the lack of your main forum page being indexed, I would give you all the standard advice - pick up some solid inbound links, submit a google sitemap, check your logs for errors, and pick up some more solid links.

Thanks for the tips.