googlebot ignoring robots.txt

**Doug Nelson** · 05-07-2007 09:32 PM

I look at my bot crawler log and I see googlebot is crawling thousands of pages that I've specifically forbidden via robots.txt. But then I remember that all those page names are being rewritten by vbseo, so how would googlebot know?

Am I correct in assuming that, for example, forbidding /forums/member.php in robots.txt is useless, because googlebot will never see a page named member.php?

**Mike** · 05-07-2007 09:35 PM

Originally Posted by Doug Nelson

Am I correct in assuming that, for example, forbidding /forums/member.php in robots.txt is useless, because googlebot will never see a page named member.php?

That is correct.

Depending on how you have your rewrites set up, you can disallow the rewritten directory, or turn off those specific rewrites so it goes back to the vbulletin default.

**Keith Cohen** · 05-07-2007 09:39 PM

Correct. If there are pages you do not wish to have indexed, turn off rewrites for those pages and exclude them via robots.txt. It'll also save a little CPU load by cutting out some un-needed rewrites.

**Doug Nelson** · 05-07-2007 10:05 PM

So it's a choice between the two? We can't have them rewritten and not indexed?

**Keith Cohen** · 05-07-2007 10:16 PM

There's really no need to rewrite URLs that will not be indexed. But, for members for example, you could just exclude /members/ assuming you're using the default rewrite rules.

**Mike** · 05-07-2007 10:16 PM

It depends on how you have them rewritten. If you rewrote them in a manner than they showed up as forum/members/member-name/, you could block it by blocking /forum/members/

**Doug Nelson** · 05-07-2007 10:20 PM

I like the rewrites for reasons beyond bot indexing. http://www.vbseo.com/members/keith-cohen/ is so much friendlier than http://www.vbseo.com/forum/member.php?u=30

**Doug Nelson** · 05-07-2007 10:26 PM

So how would I edit this section of robots.txt to take into consideration the rewrites?

Disallow: /forums/attachment.php
Disallow: /forums/calendar.php
Disallow: /forums/cron.php
Disallow: /forums/editpost.php
Disallow: /forums/global.php
Disallow: /forums/image.php
Disallow: /forums/inlinemod.php
Disallow: /forums/joinrequests.php
Disallow: /forums/login.php
Disallow: /forums/member.php
Disallow: /forums/memberlist.php
Disallow: /forums/misc.php
Disallow: /forums/moderator.php
Disallow: /forums/newattachment.php
Disallow: /forums/newreply.php
Disallow: /forums/newthread.php
Disallow: /forums/online.php
Disallow: /forums/poll.php
Disallow: /forums/postings.php
Disallow: /forums/printthread.php
Disallow: /forums/private.php
Disallow: /forums/profile.php
Disallow: /forums/register.php
Disallow: /forums/report.php
Disallow: /forums/reputation.php
Disallow: /forums/search.php
Disallow: /forums/sendmessage.php
Disallow: /forums/showgroups.php
Disallow: /forums/showpost.php
Disallow: /forums/subscription.php
Disallow: /forums/threadrate.php
Disallow: /forums/usercp.php
Disallow: /forums/usernote.php

**Mike** · 05-07-2007 10:34 PM

Disallow: /forums/members/

And you can take out: Disallow: /forums/member.php

**Doug Nelson** · 05-07-2007 10:54 PM

Is that the only one? showpost.php seems to be my big one, but I'd definitely want those kept rewritten. printthread.php is next.

**Keith Cohen** · 05-07-2007 11:08 PM

For printthread you will need to disable rewriting.

Same for showpost. And most people use the Permalink option to replace the Post Number Link with the Permalink, since individual showpost pages have never indexed well anyway.

**Code Monkey** · 05-08-2007 12:25 AM

On any page you do not want to be indexed, or you want the crawlers to be shooed away from, just add this into the <head> section of that template.

Code:


<meta name="ROBOTS" content="NOINDEX, NOFOLLOW">

**Doug Nelson** · 05-08-2007 12:42 AM

An excellent tip, thanks. But it doesn't really cut down on traffic, since the bot has to load the page to know not to index it.

**Mike** · 05-08-2007 12:55 AM

If you want to continue to rewrite posts and be able to add them to you robots.txt, you can make your showpost url rewrite custom and change it to this..

Code:

showpost/[post_id]-[post_count]

then add this to yoru robots.txt. file

Code:

Disallow: /forums/showpost/

**Code Monkey** · 05-08-2007 12:56 AM

You can also add rel="nofollow" to the links to each post.

googlebot ignoring robots.txt

LinkBack

Thread Tools

Display

googlebot ignoring robots.txt

Similar Threads

add sitemap to robots.txt for autodiscovery

quick site map question

Server Overload

Googlebot activity in the vbSEO log

Googlebot not following the sitemap?

Posting Permissions