vBulletin SEO Forums

SEO

vBulletin Search Engine Optimization

Buy vBSEO Now! HACKER SAFE certified sites prevent over 99.9% of hacker crime.
ne nw
vBSEO Total Support Team Launches DeskPro New vBSEO Discount Level for Network Builders vBSEO 3.2.0 GOLD Has Landed Success with vBSEO = 600ore Web Visitors + $1400 in a Day! Crawlability Inc. Files for SEO Technology Patent
se sw

User profile robots.txt

This is a discussion on User profile robots.txt within the Link Building forums, part of the vBulletin SEO Discussion category; I want to disable user profiles altogether from search engines, but they continue to view them constantly anyways. This is ...

Go Back   vBulletin SEO Forums > vBulletin SEO Discussion > Link Building

User profile robots.txt User profile robots.txt

Enhancing 80 million pages.

 Register FAQ Members List Social Groups Calendar Search Today's Posts Mark Forums Read

Reply

 

LinkBack Thread Tools
  #1  
Old 06-13-2007, 10:04 AM
Member
  
Real Name: Mark Prater
Join Date: May 2007
Posts: 36
User profile robots.txt
I want to disable user profiles altogether from search engines, but they continue to view them constantly anyways. This is my robots.txt


User-Agent: Googlebot-Image
Disallow: /
User-agent: *
Disallow: /attachment.php
Disallow: /faq.php
Disallow: /newattachment.php
Disallow: /avatar.php
Disallow: /forumdisplay.php?f=23
Disallow: /forumdisplay.php?f=32
Disallow: /forumdisplay.php?f=33
Disallow: /forumdisplay.php?f=39
Disallow: /members
Disallow: /editpost.php
Disallow: /login.php
Disallow: /member.php
Disallow: /member.php?
Disallow: member.php?u=
Disallow: /member2.php
Disallow: /misc.php
Disallow: /moderator.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /calendar.php
Disallow: /online.php
Disallow: /poll.php
Disallow: /postings.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /private2.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /search.php
Disallow: /sendtofriend.php
Disallow: /showgroups.php
Disallow: /showpost.php
Disallow: /threadrate.php
Disallow: /usercp.php
Disallow: /usernote.php
Disallow: /sendmessage.php
Disallow: /register.php
Disallow: /subscription.php
Disallow: /admincp/
Disallow: /archive/
Disallow: /clientscript/
Disallow: /cpstyles/
Disallow: /customavatars/
Disallow: /images/
Disallow: /includes/
Disallow: /install/
Disallow: /modcp/
Disallow: /subscriptions/
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #2  
Old 06-13-2007, 10:52 AM
Mert Gökçeimam's Avatar
vBSEO.com Webmaster
vBSEO Total Customer SupportvBulletin HackerBig Board Administrator
  
Real Name: Lizard King
Join Date: Oct 2005
Location: Istanbul
Posts: 7,883
Blog Entries: 1
Send a message via MSN to Mert Gökçeimam
Did you remove member.php files from your sitemaps ? Check vBSEO Google/Yahoo Sitemap Generator settings and make sure didn't add member profiles to your sitemaps.
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #3  
Old 06-17-2007, 08:22 PM
Member
  
Real Name: Mark Prater
Join Date: May 2007
Posts: 36
Quote:
Originally Posted by Mert Gökçeimam View Post
Did you remove member.php files from your sitemaps ? Check vBSEO Google/Yahoo Sitemap Generator settings and make sure didn't add member profiles to your sitemaps.
I dont have sitemaps.
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #4  
Old 06-17-2007, 08:48 PM
Keith Cohen's Avatar
vBSEO Staff
vBSEO Total Customer SupportBig Board Administrator
  
Real Name: Keith Cohen
Join Date: Jul 2005
Location: Raleigh, NC USA
Posts: 6,266
You should be running vBSEO Sitemap for best results.

Your ROBOTS.TXT should be fine. Just disable rewrites for user profiles in your vBSEO control panel. That will cause them to use the default vBulletin URLs, which will be blocked by your member.php entry in your ROBOTS.TXT.
__________________
Keith Cohen / Crawlability Inc.
Support Team Launches New DeskPro Powered Tool Enhanced Support at Your Service

vBSEO 3.2.0 Launched - Maximum Overdrive for Your Web Traffic! Over 100 Instant SEO Optimizations

6X Traffic - $1400 in One Day with vBSEO! Imagine What the vBSEO Patent Pending Technology Can Do For You.


My Personal Sites: My Blog | GPS Discussion Forum
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #5  
Old 06-17-2007, 10:32 PM
Dave Hybrid's Avatar
Senior Member
  
Real Name: Dave Hybrid
Join Date: Mar 2007
Location: UK
Posts: 321
Add /members/ to robots.txt as they are re-written URLs.

member.php only excludes the old URLs.
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #6  
Old 06-17-2007, 10:37 PM
Keith Cohen's Avatar
vBSEO Staff
vBSEO Total Customer SupportBig Board Administrator
  
Real Name: Keith Cohen
Join Date: Jul 2005
Location: Raleigh, NC USA
Posts: 6,266
If you aren't allowing search engines to index them, there's no need to rewrite member profiles. It's just using extra CPU that isn't necessary.
__________________
Keith Cohen / Crawlability Inc.
Support Team Launches New DeskPro Powered Tool Enhanced Support at Your Service

vBSEO 3.2.0 Launched - Maximum Overdrive for Your Web Traffic! Over 100 Instant SEO Optimizations

6X Traffic - $1400 in One Day with vBSEO! Imagine What the vBSEO Patent Pending Technology Can Do For You.


My Personal Sites: My Blog | GPS Discussion Forum
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #7  
Old 06-17-2007, 10:47 PM
Dave Hybrid's Avatar
Senior Member
  
Real Name: Dave Hybrid
Join Date: Mar 2007
Location: UK
Posts: 321
Isnt rewriting profiles enabled by default?

How do you disable it?
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #8  
Old 06-18-2007, 12:14 AM
Keith Cohen's Avatar
vBSEO Staff
vBSEO Total Customer SupportBig Board Administrator
  
Real Name: Keith Cohen
Join Date: Jul 2005
Location: Raleigh, NC USA
Posts: 6,266
Under the URL section in the vBSEO Control Panel, click the + (plus sign) to open up all of the options for customizing.
__________________
Keith Cohen / Crawlability Inc.
Support Team Launches New DeskPro Powered Tool Enhanced Support at Your Service

vBSEO 3.2.0 Launched - Maximum Overdrive for Your Web Traffic! Over 100 Instant SEO Optimizations

6X Traffic - $1400 in One Day with vBSEO! Imagine What the vBSEO Patent Pending Technology Can Do For You.


My Personal Sites: My Blog | GPS Discussion Forum
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #9  
Old 06-20-2007, 05:41 PM
Member
  
Real Name: mal
Join Date: Mar 2006
Posts: 58
Dave:
Out of interest, which spiders are crawling profiles ?

I've noticed google doesn't however the yahoo crawler does and it shows as a stop "No Permissions" error in WOL. I've all but eliminated this for the yahoo crawler by placing conditionals around certain code in a few templates. Experimental.
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #10  
Old 06-20-2007, 05:45 PM
Dave Hybrid's Avatar
Senior Member
  
Real Name: Dave Hybrid
Join Date: Mar 2007
Location: UK
Posts: 321
All spiders will spider them unless you block the via robots.txt.

Why bother adding code when one line in robots.txt stops them 100%.
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #11  
Old 06-20-2007, 06:24 PM
Member
  
Real Name: mal
Join Date: Mar 2006
Posts: 58
Are you certain about that. What if they follow a direct link from any page anywhere on the forum ? There are indeed many that link direct to profiles.


For yahoo it makes no difference if there is a rel="nofollow"
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
Reply

« almost 20 000 members | fitness/bodybuilding/powerlifting boards link exchange »
Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On

Similar Threads

Thread Thread Starter Forum Replies Last Post
Usernames with special characters linking to wrong user profile. vissa Bug Reporting 11 11-17-2005 10:04 PM


All times are GMT -4. The time now is 04:52 AM.

Contact Us - vBSEO Home - Site Map - Top

Powered by vBulletin Version 3.8.0 Release Candidate 1
Copyright ©2000 - 2008, Jelsoft Enterprises Ltd.
SEO by vBSEO 3.2.5 ©2008, Crawlability, Inc.