Google Not Indexing Threads

[dsa411]

I have had vbseo for over a year now but for some reason google is not indexing any threads. When i log into google webmaster tools its complaining that they are being blocked by robots.txt. Yahoo is indexing threads with out any issues. When I use google robots.txt tool to find out if spiders can access threads properly it reports every thing is ok. Also For some reason google is still attempting to index my vb archive as well (disabled for over a year). Here is my robots.txt file.

User-agent: *

Disallow: /amy/
Disallow: /webmail/


Disallow: /forum/ajax.php
Disallow: /forum/ajax_cron.php
Disallow: /forum/archive
Disallow: /forum/attachment.php
Disallow: /forum/checkspelling.php
Disallow: /forum/cron.php
Disallow: /forum/editpost.php
Disallow: /forum/external.php
Disallow: /forum/faq.php
Disallow: /forum/global.php
Disallow: /forum/image.php
Disallow: /forum/joinrequest.php
Disallow: /forum/login.php
Disallow: /forum/misc.php
Disallow: /forum/moderator.php
Disallow: /forum/newattatchment.php
Disallow: /forum/newreply.php
Disallow: /forum/newthread.php
Disallow: /forum/online.php
Disallow: /forum/payment_gateway.php
Disallow: /forum/payments.php
Disallow: /forum/pdfthread.php
Disallow: /forum/poll.php
Disallow: /forum/postings.php
Disallow: /forum/printthread.php
Disallow: /forum/private.php
Disallow: /forum/profile.php
Disallow: /forum/register.php
Disallow: /forum/report.php
Disallow: /forum/reputation.php
Disallow: /forum/search.php
Disallow: /forum/sendmessage.php
Disallow: /forum/subcription.php
Disallow: /forum/subcriptions.php
Disallow: /forum/threadrate.php
Disallow: /forum/usercp.php
Disallow: /forum/usernote.php

Disallow: /forum/admincp/
Disallow: /forum/images/
Disallow: /forum/modcp/
Disallow: /forum/articlebot/
Disallow: /forum/clientscript/
Disallow: /forum/customavatars/
Disallow: /forum/customprofilepics/
Disallow: /forum/files/
Disallow: /forum/install/
Disallow: /forum/cpstyles/
Disallow: /forum/images/
Disallow: /forum/includes/
Disallow: /forum/subscriptions/
Disallow: /forum/attachments/
Disallow: /forum/chat/
Disallow: /forum/chat/
Disallow: /forum/chat_auth.php
Disallow: /forum/chat.php
Disallow: /forum/chat_global.php
Disallow: /forum/vbimghost.php
Disallow: /fourm/vbseocp.php
Disallow: /forum/vbseo_sitemap
Disallow: /forum/moderation.php
Disallow: /forum/inline.php
Disallow: /forum/calendar.php
Disallow: /forum/threadprefix.php

sitemap: http://mysite.com/forum/sitemap_index.xml.gz

[Joe Ward]

Can you post us the exact error message you get and/or a screenshot?

[dsa411]

Sure here is a screenshot of google webmaster tools.

[Joe Ward]

I can't see anything wrong with your robots.txt file right now, except for this typo:

Disallow: /fourm/vbseocp.php

However - I recommend that you delete the robots.txt asap, and see if Googlebot is then able to crawl your site (which it certainly should be able to).

Once it does, you could re-add the robots.txt and try to troubleshoot.

I was thinking that there might be an error in one of the lines causing the bot to block everything in the forum directory, but I cannot see one.

Is there any extra spaces in the file after forum in any of the lines?

[Joe Ward]

Is forum f12 a guest accessible forum?

[dsa411]

Originally Posted by Joe Ward

Is forum f12 a guest accessible forum?

Yes

[dsa411]

Originally Posted by Joe Ward

I can't see anything wrong with your robots.txt file right now, except for this typo:

Disallow: /fourm/vbseocp.php

However - I recommend that you delete the robots.txt asap, and see if Googlebot is then able to crawl your site (which it certainly should be able to).

Once it does, you could re-add the robots.txt and try to troubleshoot.

I was thinking that there might be an error in one of the lines causing the bot to block everything in the forum directory, but I cannot see one.

Is there any extra spaces in the file after forum in any of the lines?

Doesn't seem like there are any extra spaces. Its weird because if i test with googles robots analysis tool everything checks out. If i try to test a line thats blocked it does say its blocked so every thing with the file seems ok. I will remove it anyway for test purposes.

[Joe Ward]

Yes - remove it and see if the indexing begins. That way we will at least know that it is some problem with the robots.txt.

[dsa411]

Originally Posted by Joe Ward

Yes - remove it and see if the indexing begins. That way we will at least know that it is some problem with the robots.txt.

I just checked Google Webmaster tools and I noticed the urls restriced by robots number has gone down from 4675 to 2919. I got a couple errors on the 12th and not one since. I also can see that some new pages are being indexed in google.

[Joe Ward]

Wait until it gets to 0 errors, before considering re-attempting a robots.txt file. However, you may well find it is unnecessary.