Sitemap generating a lot more urls

poprulz · #1 04-22-2006, 04:49 PM

I have got a problem here with vBSEO sitemap generator.

this my forum stats.
Members: 3,402
Threads: 3,151
Posts: 15,883

So i guess the urls shouldn't be more than 24,000 (rough). But this is wot i get in sitemap report.
Date 2006-04-22 07:00
Processing time 743.82 s
Total URLs 46,655 (+2)
Forumdisplay URLs 494 (-)
Showthread URLs 23,208 (-)
ShowPost URLs 15,844 (-)
Archive URLs 267 (-)
Member Profile URLs 6,776 (+2)
Poll Results URLs 64 (-)

There is a huge difference in showthread and almost double in member profile urls.

I had a look at the xml file and i guess the non SEOed urls r been added along with SEOed urls.

any idea?

Oleg Ignatiuk · #2 04-22-2006, 05:09 PM

Hello,

if you are using a Sitemap generator v1.7, there is an option to include original vB URLs along with SEOed URLs in the sitemap.

poprulz · #3 04-22-2006, 05:20 PM

ohh.. gotit!

thanx Oleg for the info

Monique · #4 06-07-2006, 04:17 AM

Hello tutti

I have a problem with "member profile"

Pratically all my pages that Google or Yahoo take into account are "member profile"

Is it possible to avoid this ?

Thanh you

Monique

Oleg Ignatiuk · #5 06-07-2006, 06:30 AM

Hello Monique,

do you mean you want to exclude member profile pages from indexing by search engines? (you can do that using robots.txt)

Monique · #6 06-07-2006, 07:54 AM

Quote:

Originally Posted by Oleg Ignatiuk

Hello Monique,

do you mean you want to exclude member profile pages from indexing by search engines?

Yes

Quote:

Originally Posted by Oleg Ignatiuk

Hello Monique,

(you can do that using robots.txt)

OK
you said also :

Quote:

Originally Posted by Oleg Ignatiuk

Hello,

your robots.txt entries are correct.
By the way, there is a new robots.txt tool available in the Google sitemaps account.

my question :

Must I have to use the Google robots.txt ?

Ca I use something like that :

# Disallow directory
User-agent: *
Disallow: /admincp/
Disallow: /ajax.php
Disallow: /articlebot/
Disallow: /attachment.php
Disallow: /attachments/
Disallow: /clientscript/
Disallow: /cpstyles/
Disallow: /customavatars/
Disallow: /customprofilepics/
Disallow: /calendar.php
Disallow: /cgi-bin/
Disallow: /cron.php
Disallow: /editpost.php
Disallow: /external.php
Disallow: /frm_attach
Disallow: /images/
Disallow: /includes/
Disallow: /inlinemod.php
Disallow: /install/
Disallow: /joinrequests.php
Disallow: /login.php
Disallow: /member.php?
Disallow: /memberlist.php
Disallow: /misc.php
Disallow: /modcp/
Disallow: /moderator.php
Disallow: /modules/
Disallow: /newattachment.php
Disallow: /newreply.php
Disallow: /newrs.cal
Disallow: /newthread.php
Disallow: /oldindex.html
Disallow: /online.php
Disallow: /poll.php
Disallow: /postings.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /profile.php
Disallow: /register.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /search.php
Disallow: /sendmessage.php
Disallow: /showgroups.php
Disallow: /ssm.js

In this forum someone said that I have to put this robots.txt in Root ? Where is it ?

There is a lot of parameters (in vBulletin, in VBSEO, in Google SiteMap) that forgot my latin

... et I have difficulties to write good english (french speaking)

Thanks

Monique

Oleg Ignatiuk · #7 06-07-2006, 10:34 AM

Hello Monique,

these robots.txt entries are correct, but only when vBulletin resides in the domain root. In your case, you should put /forum into every line, like:

Code:

User-agent: *
Disallow: /forum/admincp/
Disallow: /forum/ajax.php
Disallow: /forum/articlebot/
... etc

and put this robots.txt in the sites root.

Monique · #8 06-07-2006, 01:53 PM

Quote:

Originally Posted by Oleg Ignatiuk

Hello Monique,

Code:

User-agent: *
Disallow: /forum/admincp/
Disallow: /forum/ajax.php
Disallow: /forum/articlebot/
... etc

and put this robots.txt in the sites root.

Thank You Oleg

so I put my robots.txt (like above /forum/admincp, etc...) in www and not in the directory forum ? is'n it ?

http://www.cameravideo.net/robots.txt

Is it OK ?

Thank you

Monique

Oleg Ignatiuk · #9 06-07-2006, 02:20 PM

Yes, exactly

If you want to exlcude your member profiles pages from indexing, you should add one more line in robots.txt:

Code:

Disallow: /forum/members/

However, you may want to keep them allowed to search engines - sometimes there is a useful information included by users in profile fields.

Rik Brown · #10 02-28-2007, 04:24 PM

Quote:

Originally Posted by Oleg Ignatiuk

Hello,

if you are using a Sitemap generator v1.7, there is an option to include original vBulletin URLs along with SEOed URLs in the sitemap.

This is happening to me with v1.7. But I'm not sure which option to change. Could you please advise the option name?

Thanks. -- Rik

Oleg Ignatiuk · #11 02-28-2007, 04:47 PM

The option's title is: "If you have vBSEO, include old urls?" (2nd option in the list)

Rik Brown · #12 02-28-2007, 04:57 PM

Quote:

Originally Posted by Oleg Ignatiuk

The option's title is: "If you have vBSEO, include old urls?" (2nd option in the list)

Oleg:

I've got that option set to "No."

FYI, I'm getting over 850,000 URLs in the Google sitemap but I only have 680,000 forum pages. Here are my other "include" settings:

Include Show Post Pages: Yes
Include Member Profile Pages: No
Include Archive Pages: No
Include Show Thread Pages: Yes
Include Forum Display Pages: Yes
Include Poll Results Pages: No

Could one of the above settings be the culprit?

Thanks. -- Rik

Oleg Ignatiuk · #13 02-28-2007, 05:12 PM

You can find exact numbers per "page type" in the generator reports.

Rik Brown · #14 02-28-2007, 05:15 PM

Quote:

Originally Posted by Oleg Ignatiuk

You can find exact numbers per "page type" in the generator reports.

Oleg: Will check there. Thanks. -- Rik

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Google Sitemap Error (Location?)	Spitfire	General Discussion	5	05-30-2006 07:12 AM
Memory Problem:	TraumTeam	Troubleshooting	1	04-06-2006 09:24 AM
Google Sitemap-Listing of old url's option-from 1.7	T2DMan	Member Articles	0	04-02-2006 09:46 AM
vBSEO Google/Yahoo Sitemap Generator for vBulletin 3.6.x, vBulletin 3.5.x & vBulletin 3.0.x	Joe Ward	General Discussion	0	12-17-2005 02:11 PM
Sitemap generator, bad urls...	psico	Bug Reporting	1	11-14-2005 10:16 AM

SEO

Sitemap generating a lot more urls

Sitemap generating a lot more urls

Similar Threads