please validate my robots.txt

Code:

User-agent: gigabot
Disallow: /

User-agent: *
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /private.php
Disallow: /printthread.php
Disallow: /sendmessage.php
Disallow: /showpost.php

is this correct ?
lot of content is indexed in serps with those filenames which leads to duplicate content on same site and lot of error messages generated.

Hello,

your robots.txt entries are correct.
By the way, there is a new robots.txt tool available in the Google sitemaps account.

Are these kind of entries still needed when using VBSEO?

Chris

Hello Chris,

yes, since these URLs are not rewritten, the robots.txt exclusions are still valid.

Thanks. Just updated mine after reading your reply.

Christoph

Originally Posted by Oleg Ignatiuk

Hello Chris,

yes, since these URLs are not rewritten, the robots.txt exclusions are still valid.

What does excluding these URLS actually do? Why would you want to do this?

Hello Josh,

this prevents SE from indexing the "noise" pages, focusing the bots on the more valuable content and reducing the load on your server.

OK, I changed mine to:

User-agent: *
Disallow: /forum/calendar.php
Disallow: /forum/editpost.php
Disallow: /forum/memberlist.php
Disallow: /forum/misc.php
Disallow: /forum/newreply.php
Disallow: /forum/newthread.php
Disallow: /forum/private.php
Disallow: /forum/register.php
Disallow: /forum/report.php
Disallow: /forum/search.php
Disallow: /forum/showgroups.php
Disallow: /forum/usercp.php
Disallow: /forum/admincp/
Disallow: /forum/modcp/
Disallow: /forum/online.php
Disallow: /forum/subscription.php
Disallow: /forum/sendtofriend.php
Disallow: /forum/threadrate.php
Disallow: /forum/attachment.php
Disallow: /forum/avatar.php
Disallow: /forum/faq.php
Disallow: /forum/usercp.php
Disallow: /forum/profile.php
Disallow: /users/
Disallow: /gallery/
Disallow: /banner/

Any problems with doing this?

Hello Josh,

these pages may have some content important to be indexed: calendar.php, showgroups.php, faq.php. Make sure that it's ok to exclude them at your forums.
Also, I'm sure it's better to index your /gallery/ (and probably /users/) folders.

As a matter of interest, where exactly is the correct place to install the robots.txt file. Each directory or only in the root?

Only in the root. You can define specific paths to block though, as seen in the previous posts in this thread.

I use the following on my site.

Code:

User-agent: gigabot
Disallow: /

User-agent: *
Disallow: /cgi-bin/
Disallow: /admincp/
Disallow: /includes/
Disallow: /install/
Disallow: /ioncube/
Disallow: /mint/
Disallow: /modcp/
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /private.php
Disallow: /printthread.php
Disallow: /sendmessage.php
Disallow: /showpost.php

Originally Posted by Oleg Ignatiuk

Also, I'm sure it's better to index your /gallery/ (and probably /users/) folders.

If I don't want spiders to crawl my member profiles, should I block the users, members and profile.php?

Originally Posted by trana

If I don't want spiders to crawl my member profiles, should I block the users, members and profile.php?

To disallow crawling of members info, you should include:

Code:

Disallow: /member.php
Disallow: /members/

(assuming your member profile URL format defined in vbseocp is "members/...")