Google can't crawl my forums?

[Corporate Factors]

I run a VB Bulliten board.

I keep getting this error that google can't crawl my Corporate Factors - Business & Venture Capital Forums because a robot.txt file is blocking the crawl.

Is this a setting in my vb software or some other error. How do I fix this?

[Michael Biddle]

What is the contents of the robot.txt file located in your root directory?

[Corporate Factors]

wow that was quick.. Can you tell me where I find this so I can show you..

[Michael Biddle]

It would be yoursite.com/robot.txt

[Corporate Factors]

I still can't find what your asking. I have vbseo can I generate a report off that or something. For whatever reason google is not crawling in my fourms every topic is failing saying it's not google friendly.

[Michael Biddle]

I am seeing your robots.txt file as:

User-agent: *
Disallow: /forum/ajax.php
Disallow: /forum/attachment.php
Disallow: /forum/calendar.php
Disallow: /forum/cron.php
Disallow: /forum/editpost.php
Disallow: /forum/global.php
Disallow: /forum/image.php
Disallow: /forum/inlinemod.php
Disallow: /forum/joinrequests.php
Disallow: /forum/login.php
Disallow: /forum/member.php
Disallow: /forum/memberlist.php
Disallow: /forum/misc.php
Disallow: /forum/moderator.php
Disallow: /forum/newattachment.php
Disallow: /forum/newreply.php
Disallow: /forum/newthread.php
Disallow: /forum/online.php
Disallow: /forum/poll.php
Disallow: /forum/postings.php
Disallow: /forum/printthread.php
Disallow: /private.php
Disallow: /forum/profile.php
Disallow: /forum/register.php
Disallow: /forum/report.php
Disallow: /forum/reputation.php
Disallow: /forum/search.php
Disallow: /forum/sendmessage.php
Disallow: /forum/showgroups.php
Disallow: /forum/subscription.php
Disallow: /forum/threadrate.php
Disallow: /forum/usercp.php
Disallow: /forum/usernote.php

None of that looks wrong, what is the exact error google gives you?

[Corporate Factors]

URL restricted by robots.txt

[briansol]

is it just certain pages? sounds like you're blocking something, but including it in the sitemap.

[Corporate Factors]

So is that I setting I need to change in admincp.. Brian ?

[briansol]

it would depend on the actual error google is giving you. Look for a trend.
are most the pages that it can't find showpost.php pages?

[Corporate Factors]

well most of the links appear to be failed vbcalander pages all I really care about is that the individual forum topics and the text therein will get crawled. error below:

http://www.corporatefactors.net/forum/cron.php?rand=421512ReferrerURL restricted by robots.txt Help

Feb 5, 2009http://www.corporatefactors.net/forum/forumdisplay.php?f=subsURL restricted by robots.txt Help

Feb 7, 2009http://www.corporatefactors.net/forum/...isplay.php?s=&daysprune=&f=subsURL restricted by robots.txt Help

Feb 7, 2009http://www.corporatefactors.net/forum/login.php?do=lostpwURL restricted by robots.txt Help

Jan 28, 2009http://www.corporatefactors.net/forum/member.php?u=6

[briansol]

Those links are fine to be blocked... since you have vbseo now, those pages don't "exist" anymore, and they SHOULD 404.

ie, the last link is now something like /forum/members/user-name-123/ instead of member.php?u=123