vBulletin SEO Forums

SEO

vBulletin Search Engine Optimization

Buy vBSEO Now! HACKER SAFE certified sites prevent over 99.9% of hacker crime.
ne nw
New vBSEO Discount Level for Network Builders Meet vBSEO Team in New York (Nov. 3rd & 4th) vBSEO 3.2.0 GOLD Has Landed Success with vBSEO = 600ore Web Visitors + $1400 in a Day! Crawlability Inc. Files for SEO Technology Patent
se sw

Sitemap generating a lot more urls

This is a discussion on Sitemap generating a lot more urls within the Troubleshooting forums, part of the vBSEO Google/Yahoo Sitemap category; I have got a problem here with vBSEO sitemap generator. this my forum stats. Members: 3,402 Threads: 3,151 Posts: 15,883 ...

Go Back   vBulletin SEO Forums > vBSEO Google/Yahoo Sitemap > Troubleshooting

Enhancing 80 million pages.

Register FAQ Members List Social Groups Calendar Search Today's Posts Mark Forums Read
  #1  
Old 04-22-2006, 04:49 PM
Junior Member
 
Join Date: Nov 2005
Posts: 24
Sitemap generating a lot more urls

I have got a problem here with vBSEO sitemap generator.

this my forum stats.
Members: 3,402
Threads: 3,151
Posts: 15,883

So i guess the urls shouldn't be more than 24,000 (rough). But this is wot i get in sitemap report.
Date 2006-04-22 07:00
Processing time 743.82 s
Total URLs 46,655 (+2)
Forumdisplay URLs 494 (-)
Showthread URLs 23,208 (-)
ShowPost URLs 15,844 (-)
Archive URLs 267 (-)
Member Profile URLs 6,776 (+2)
Poll Results URLs 64 (-)

There is a huge difference in showthread and almost double in member profile urls.

I had a look at the xml file and i guess the non SEOed urls r been added along with SEOed urls.

any idea?
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #2  
Old 04-22-2006, 05:09 PM
Oleg Ignatiuk's Avatar
vBSEO Staff
vBSEO Total Customer SupportvBSEO Documenter
 
Real Name: Oleg Ignatiuk
Join Date: Jun 2005
Location: Belarus
Posts: 21,450
Blog Entries: 1
Re: Sitemap generating a lot more urls

Hello,

if you are using a Sitemap generator v1.7, there is an option to include original vB URLs along with SEOed URLs in the sitemap.
__________________
Oleg Ignatiuk / Crawlability Inc.
vBSEO 3.2.0 Launched - Maximum Overdrive for Your Web Traffic! Over 100 Instant SEO Optimizations

vBSEO Google Sitemap Generator - Version 2.2 Released Mandatory Upgrade for vBSEO 3.2.0 GOLD

6X Traffic - $1400 in One Day with vBSEO! Imagine What the vBSEO Patent Pending Technology Can Do For You.

Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #3  
Old 04-22-2006, 05:20 PM
Junior Member
 
Join Date: Nov 2005
Posts: 24
Re: Sitemap generating a lot more urls

ohh.. gotit!

thanx Oleg for the info
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #4  
Old 06-07-2006, 04:17 AM
Junior Member
 
Join Date: Jan 2006
Posts: 23
Re: Sitemap generating a lot more urls

Hello tutti

I have a problem with "member profile"

Pratically all my pages that Google or Yahoo take into account are "member profile"



Is it possible to avoid this ?

Thanh you

Monique
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #5  
Old 06-07-2006, 06:30 AM
Oleg Ignatiuk's Avatar
vBSEO Staff
vBSEO Total Customer SupportvBSEO Documenter
 
Real Name: Oleg Ignatiuk
Join Date: Jun 2005
Location: Belarus
Posts: 21,450
Blog Entries: 1
Re: Sitemap generating a lot more urls

Hello Monique,

do you mean you want to exclude member profile pages from indexing by search engines? (you can do that using robots.txt)
__________________
Oleg Ignatiuk / Crawlability Inc.
vBSEO 3.2.0 Launched - Maximum Overdrive for Your Web Traffic! Over 100 Instant SEO Optimizations

vBSEO Google Sitemap Generator - Version 2.2 Released Mandatory Upgrade for vBSEO 3.2.0 GOLD

6X Traffic - $1400 in One Day with vBSEO! Imagine What the vBSEO Patent Pending Technology Can Do For You.

Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #6  
Old 06-07-2006, 07:54 AM
Junior Member
 
Join Date: Jan 2006
Posts: 23
Re: Sitemap generating a lot more urls

Quote:
Originally Posted by Oleg Ignatiuk
Hello Monique,

do you mean you want to exclude member profile pages from indexing by search engines?
Yes

Quote:
Originally Posted by Oleg Ignatiuk
Hello Monique,

(you can do that using robots.txt)
OK
you said also :

Quote:
Originally Posted by Oleg Ignatiuk
Hello,

your robots.txt entries are correct.
By the way, there is a new robots.txt tool available in the Google sitemaps account.
my question :

Must I have to use the Google robots.txt ?

Ca I use something like that :

# Disallow directory
User-agent: *
Disallow: /admincp/
Disallow: /ajax.php
Disallow: /articlebot/
Disallow: /attachment.php
Disallow: /attachments/
Disallow: /clientscript/
Disallow: /cpstyles/
Disallow: /customavatars/
Disallow: /customprofilepics/
Disallow: /calendar.php
Disallow: /cgi-bin/
Disallow: /cron.php
Disallow: /editpost.php
Disallow: /external.php
Disallow: /frm_attach
Disallow: /images/
Disallow: /includes/
Disallow: /inlinemod.php
Disallow: /install/
Disallow: /joinrequests.php
Disallow: /login.php
Disallow: /member.php?
Disallow: /memberlist.php
Disallow: /misc.php
Disallow: /modcp/
Disallow: /moderator.php
Disallow: /modules/
Disallow: /newattachment.php
Disallow: /newreply.php
Disallow: /newrs.cal
Disallow: /newthread.php
Disallow: /oldindex.html
Disallow: /online.php
Disallow: /poll.php
Disallow: /postings.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /profile.php
Disallow: /register.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /search.php
Disallow: /sendmessage.php
Disallow: /showgroups.php
Disallow: /ssm.js

In this forum someone said that I have to put this robots.txt in Root ? Where is it ?

There is a lot of parameters (in vBulletin, in VBSEO, in Google SiteMap) that forgot my latin ... et I have difficulties to write good english (french speaking)

Thanks

Monique
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #7  
Old 06-07-2006, 10:34 AM
Oleg Ignatiuk's Avatar
vBSEO Staff
vBSEO Total Customer SupportvBSEO Documenter
 
Real Name: Oleg Ignatiuk
Join Date: Jun 2005
Location: Belarus
Posts: 21,450
Blog Entries: 1
Re: Sitemap generating a lot more urls

Hello Monique,

these robots.txt entries are correct, but only when vBulletin resides in the domain root. In your case, you should put /forum into every line, like:
Code:
User-agent: *
Disallow: /forum/admincp/
Disallow: /forum/ajax.php
Disallow: /forum/articlebot/
... etc
and put this robots.txt in the sites root.
__________________
Oleg Ignatiuk / Crawlability Inc.
vBSEO 3.2.0 Launched - Maximum Overdrive for Your Web Traffic! Over 100 Instant SEO Optimizations

vBSEO Google Sitemap Generator - Version 2.2 Released Mandatory Upgrade for vBSEO 3.2.0 GOLD

6X Traffic - $1400 in One Day with vBSEO! Imagine What the vBSEO Patent Pending Technology Can Do For You.

Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #8  
Old 06-07-2006, 01:53 PM
Junior Member
 
Join Date: Jan 2006
Posts: 23
Re: Sitemap generating a lot more urls

Quote:
Originally Posted by Oleg Ignatiuk
Hello Monique,

Code:
User-agent: *
Disallow: /forum/admincp/
Disallow: /forum/ajax.php
Disallow: /forum/articlebot/
... etc
and put this robots.txt in the sites root.
Thank You Oleg

so I put my robots.txt (like above /forum/admincp, etc...) in www and not in the directory forum ? is'n it ?

http://www.cameravideo.net/robots.txt

Is it OK ?

Thank you

Monique
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #9  
Old 06-07-2006, 02:20 PM
Oleg Ignatiuk's Avatar
vBSEO Staff
vBSEO Total Customer SupportvBSEO Documenter
 
Real Name: Oleg Ignatiuk
Join Date: Jun 2005
Location: Belarus
Posts: 21,450
Blog Entries: 1
Re: Sitemap generating a lot more urls

Yes, exactly

If you want to exlcude your member profiles pages from indexing, you should add one more line in robots.txt:
Code:
Disallow: /forum/members/
However, you may want to keep them allowed to search engines - sometimes there is a useful information included by users in profile fields.
__________________
Oleg Ignatiuk / Crawlability Inc.
vBSEO 3.2.0 Launched - Maximum Overdrive for Your Web Traffic! Over 100 Instant SEO Optimizations

vBSEO Google Sitemap Generator - Version 2.2 Released Mandatory Upgrade for vBSEO 3.2.0 GOLD

6X Traffic - $1400 in One Day with vBSEO! Imagine What the vBSEO Patent Pending Technology Can Do For You.

Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #10  
Old 02-28-2007, 04:24 PM
Junior Member
 
Real Name: Rik Brown
Join Date: Apr 2006
Location: St. Louis, Missouri, USA
Posts: 17
Quote:
Originally Posted by Oleg Ignatiuk View Post
Hello,

if you are using a Sitemap generator v1.7, there is an option to include original vBulletin URLs along with SEOed URLs in the sitemap.
This is happening to me with v1.7. But I'm not sure which option to change. Could you please advise the option name?

Thanks. -- Rik
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #11  
Old 02-28-2007, 04:47 PM
Oleg Ignatiuk's Avatar
vBSEO Staff
vBSEO Total Customer SupportvBSEO Documenter
 
Real Name: Oleg Ignatiuk
Join Date: Jun 2005
Location: Belarus
Posts: 21,450
Blog Entries: 1
The option's title is: "If you have vBSEO, include old urls?" (2nd option in the list)
__________________
Oleg Ignatiuk / Crawlability Inc.
vBSEO 3.2.0 Launched - Maximum Overdrive for Your Web Traffic! Over 100 Instant SEO Optimizations

vBSEO Google Sitemap Generator - Version 2.2 Released Mandatory Upgrade for vBSEO 3.2.0 GOLD

6X Traffic - $1400 in One Day with vBSEO! Imagine What the vBSEO Patent Pending Technology Can Do For You.

Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #12  
Old 02-28-2007, 04:57 PM
Junior Member
 
Real Name: Rik Brown
Join Date: Apr 2006
Location: St. Louis, Missouri, USA
Posts: 17
Quote:
Originally Posted by Oleg Ignatiuk View Post
The option's title is: "If you have vBSEO, include old urls?" (2nd option in the list)
Oleg:

I've got that option set to "No."

FYI, I'm getting over 850,000 URLs in the Google sitemap but I only have 680,000 forum pages. Here are my other "include" settings:

Include Show Post Pages: Yes
Include Member Profile Pages: No
Include Archive Pages: No
Include Show Thread Pages: Yes
Include Forum Display Pages: Yes
Include Poll Results Pages: No

Could one of the above settings be the culprit?

Thanks. -- Rik
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #13  
Old 02-28-2007, 05:12 PM
Oleg Ignatiuk's Avatar
vBSEO Staff
vBSEO Total Customer SupportvBSEO Documenter
 
Real Name: Oleg Ignatiuk
Join Date: Jun 2005
Location: Belarus
Posts: 21,450
Blog Entries: 1
You can find exact numbers per "page type" in the generator reports.
__________________
Oleg Ignatiuk / Crawlability Inc.
vBSEO 3.2.0 Launched - Maximum Overdrive for Your Web Traffic! Over 100 Instant SEO Optimizations

vBSEO Google Sitemap Generator - Version 2.2 Released Mandatory Upgrade for vBSEO 3.2.0 GOLD

6X Traffic - $1400 in One Day with vBSEO! Imagine What the vBSEO Patent Pending Technology Can Do For You.

Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #14  
Old 02-28-2007, 05:15 PM
Junior Member
 
Real Name: Rik Brown
Join Date: Apr 2006
Location: St. Louis, Missouri, USA
Posts: 17
Quote:
Originally Posted by Oleg Ignatiuk View Post
You can find exact numbers per "page type" in the generator reports.
Oleg: Will check there. Thanks. -- Rik
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads

Thread Thread Starter Forum Replies Last Post
Google Sitemap Error (Location?) Spitfire General Discussion 5 05-30-2006 07:12 AM
Memory Problem: TraumTeam Troubleshooting 1 04-06-2006 09:24 AM
Google Sitemap-Listing of old url's option-from 1.7 T2DMan Member Articles 0 04-02-2006 09:46 AM
vBSEO Google/Yahoo Sitemap Generator for vBulletin 3.6.x, vBulletin 3.5.x & vBulletin 3.0.x Joe Ward General Discussion 0 12-17-2005 02:11 PM
Sitemap generator, bad urls... psico Bug Reporting 1 11-14-2005 10:16 AM


All times are GMT -4. The time now is 05:17 PM.


Powered by vBulletin Version 3.8.0 Beta 3
Copyright ©2000 - 2008, Jelsoft Enterprises Ltd.
SEO by vBSEO 3.2.5 ©2008, Crawlability, Inc.