vBulletin SEO Forums

SEO

vBulletin Search Engine Optimization

Buy vBSEO Now! HACKER SAFE certified sites prevent over 99.9% of hacker crime.
ne nw
New vBSEO Discount Level for Network Builders Meet vBSEO Team in New York (Nov. 3rd & 4th) vBSEO 3.2.0 GOLD Has Landed Success with vBSEO = 600ore Web Visitors + $1400 in a Day! Crawlability Inc. Files for SEO Technology Patent
se sw

SE Bots Activity Details,crawling members!

This is a discussion on SE Bots Activity Details,crawling members! within the Troubleshooting forums, part of the vBSEO Google/Yahoo Sitemap category; Hi There i was looking at my SE Bots Activity Details today and i noticed that Googlebot still crawling members ...

Go Back   vBulletin SEO Forums > vBSEO Google/Yahoo Sitemap > Troubleshooting

Enhancing 80 million pages.

Register FAQ Members List Social Groups Calendar Search Today's Posts Mark Forums Read
  #1  
Old 04-10-2008, 04:10 AM
Senior Member
 
Real Name: Seleno
Join Date: Mar 2007
Posts: 211
Blog Entries: 1
Exclamation SE Bots Activity Details,crawling members!

Hi There
i was looking at my SE Bots Activity Details today and i noticed that Googlebot still crawling members

member.php (Googlebot show 445,yahoo show172)
faq.php (Googlebot 12)
also i found ia_archiver and mediapartners

so how can i keep GoogleBot away from member.php,faq.php
and also ia_archiver
should i add this to robots.txt?

for members i have nofollow
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #2  
Old 04-11-2008, 01:36 AM
Joe Ward's Avatar
vBSEO Staff
vBSEO Total Customer SupportvBSEO Documenter
 
Real Name: Joseph Ward
Join Date: Jun 2005
Location: Puerto Rico
Posts: 19,320
Blog Entries: 7
robots.txt will work.
__________________
Joe Ward / Crawlability Inc.
vBSEO 3.2.0 Launched - Maximum Overdrive for Your Web Traffic! Over 100 Instant SEO Optimizations

vBSEO Google Sitemap Generator - Version 2.2 Released Mandatory Upgrade for vBSEO 3.2.0 GOLD

6X Traffic - $1400 in One Day with vBSEO! Imagine What the vBSEO Patent Pending Technology Can Do For You.

Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #3  
Old 04-11-2008, 02:38 AM
Senior Member
 
Real Name: Seleno
Join Date: Mar 2007
Posts: 211
Blog Entries: 1
thanx Joe Ward
if i prevent googlebot to visit member profile googlebot will crawl threads more...
right?
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #4  
Old 04-12-2008, 12:05 AM
Joe Ward's Avatar
vBSEO Staff
vBSEO Total Customer SupportvBSEO Documenter
 
Real Name: Joseph Ward
Join Date: Jun 2005
Location: Puerto Rico
Posts: 19,320
Blog Entries: 7
Yes - that's correct. If you eliminate large amounts of content, then it allows Googlebot to focus on the remaining threads.
__________________
Joe Ward / Crawlability Inc.
vBSEO 3.2.0 Launched - Maximum Overdrive for Your Web Traffic! Over 100 Instant SEO Optimizations

vBSEO Google Sitemap Generator - Version 2.2 Released Mandatory Upgrade for vBSEO 3.2.0 GOLD

6X Traffic - $1400 in One Day with vBSEO! Imagine What the vBSEO Patent Pending Technology Can Do For You.

Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #5  
Old 04-12-2008, 12:10 AM
Joe Ward's Avatar
vBSEO Staff
vBSEO Total Customer SupportvBSEO Documenter
 
Real Name: Joseph Ward
Join Date: Jun 2005
Location: Puerto Rico
Posts: 19,320
Blog Entries: 7
Here is some feedback on Juan related to keeping member profiles open to search engines:
Disable the Printable version to avoid duplicate?

I weighted in briefly here:
Quality of the crawl
__________________
Joe Ward / Crawlability Inc.
vBSEO 3.2.0 Launched - Maximum Overdrive for Your Web Traffic! Over 100 Instant SEO Optimizations

vBSEO Google Sitemap Generator - Version 2.2 Released Mandatory Upgrade for vBSEO 3.2.0 GOLD

6X Traffic - $1400 in One Day with vBSEO! Imagine What the vBSEO Patent Pending Technology Can Do For You.

Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
Reply

Tags
crawling, google sitemap, googlebot, se bots

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads

Thread Thread Starter Forum Replies Last Post
SE Bots Activity 7ayr.net General Discussion 11 12-04-2007 10:49 AM
Bots Activity Log Not show anything! alaali Troubleshooting 3 10-15-2007 05:28 PM
SE Bots Activity Log Itworx4me Troubleshooting 12 11-26-2006 02:29 PM
SE Bots Activity Log Sonnie Troubleshooting 8 09-30-2006 04:48 PM
SE Bots Activity Details fsb-holger Bug Reporting 3 11-22-2005 10:35 AM


All times are GMT -4. The time now is 04:13 PM.


Powered by vBulletin Version 3.8.0 Beta 3
Copyright ©2000 - 2008, Jelsoft Enterprises Ltd.
SEO by vBSEO 3.2.5 ©2008, Crawlability, Inc.