Quality of the crawl

More questions!

Okay, is it worse if the bots are crawling profiles but not showpost...If I had a choice between the two, wouldn't I want showpost?

If so, any positive effect to nofollow profiles for a while? Say that google sends 150 bots in a day to my site. Am I wasting some of the eyes on time I am getting from googlebot on profiles? Then later, after they crawl and index my other pages, to turn them back on to profiles. Or am I thinking about this in the wrong way? Or is this part of the mystery of google...and no one knows the answer.

Thanks for your thoughts.

you don't want showpost because it's duplicate content to showthread. the profiles are unique content (more or less).

Personally, I would not want the bots crawling either type of page: member profile or showpost.

Of course, they are none to generate some traffic. Juan also pointed out that they generate traffic from "vanity searches".

It depends on your personal strategy. For me, leaving them out would be preferred, especially if you already have a very content rich site.

Okay, this is my robots.txt

User-agent: *
Disallow: /ajax.php
Disallow: /attachment.php
Disallow: /calendar.php
Disallow: /cron.php
Disallow: /editpost.php
Disallow: /global.php
Disallow: /image.php
Disallow: /faq.php
Disallow: /inlinemod.php
Disallow: /joinrequests.php
Disallow: /login.php
Disallow: /misc.php
Disallow: /moderator.php
Disallow: /newattachment.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /poll.php
Disallow: /postings.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /profile.php
Disallow: /register.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /search.php
Disallow: /sendmessage.php
Disallow: /payments.php
Disallow: /showpost.php
Disallow: /showgroups.php
Disallow: /spiders.php
Disallow: /subscription.php
Disallow: /threadrate.php
Disallow: /usercp.php
Disallow: /usernote.php
Disallow: /admincp/
Disallow: /cgi-bin/
Disallow: /includes/
Disallow: /install/
Disallow: /modcp/
Disallow: /customavatars/
Disallow: /xmlrpc.php
Disallow: /archive/
Disallow: /sitemap/

I am a bit confused because vBSEO Sitemap Reports shows bots crawling member profiles but not showpost. Am I messed up somehow? Or will they crawl, but ignore?

The above is blocking the old dynamic format. To use robots.txt to block member profiles, you would have to rewrite your profiles into a sub-directory such as "/members/" and then block that full directory.

The search engines are not finding links to profile.php anymore since it is rewritten.

Joe,

I set myself up for laughs at my expense with the title to this post. But is there any guide on how I would do that?

Within the vBSEO control panel (vbseocp.php), you can configure your member profile urls to include the "/members/" virtual directory component.

Once that is setup and working correctly, you can simply add:

Code:

 Disallow: /members/

to your robots.txt file.

Joe,

Thanks! Done and working. I will let you all know if this is what pushes me to PR 10

Or PR 11.