What is being included/excluded

theguru · #1 04-22-2007, 03:06 PM

Before upgrading to SiteMap 2.0, my emailed stats showed

Quote:

Summary:
============================
Forum Display: 41
Show Thread: 1112
Show Post: 6866
Member Profiles: 17907
Archive: 12

Total Indexed URLs: 25939
Total Processing Time: 28.14 seconds

Google ping: Successful.

============================

but now I am getting

Quote:

Summary:
============================
Forum Display: 0
Show Thread:
Show Post:
Member Profiles: 35878
Poll Results: 0
Archive: 1

Total Indexed URLs: 35880
Total Processing Time: 8.43 seconds

Google ping: Successful.
Yahoo ping: Successful.

============================

So what I need to find out is why are the forums, posts & polls now not being included and why our member profiles have doubled?

Keith Cohen · #2 04-22-2007, 03:12 PM

Your threads and posts aren't included because guests don't have access to view them. You have your permissions set to only allow guests to see thread titles, so vBSEO Sitemap no longer includes those URLs since they would just be indexing a "Login" page for all of them.

For your member profiles, you may have the option enabled that causes vBSEO Sitemap to include the rewritten and non-rewritten versions of URLs, to help the search engines get the old URLs out of their indexes. (They'll see the 301 redirect and will eventually remove the old URLs.)

theguru · #3 04-22-2007, 07:40 PM

Thanks Keith.

Yes, we do require users to log in before being able to view thread contents, but spiders are setup as regular users therefore the actual page should get indexed and not a login page.

Also, guests do not have permissions to view member profiles, so using what you have described, why are those being included?

What I would like to be able to do is have the sitemap created using the permissions of a specific group or groups. Is this possible please?

Keith Cohen · #4 04-22-2007, 07:48 PM

Quote:

Originally Posted by theguru

Yes, we do require users to log in before being able to view thread contents, but spiders are setup as regular users therefore the actual page should get indexed and not a login page.

This is considered blackhat SEO, so we do not wish to support that type of thing. Guests need to be able to see the same thing that search engine spiders see.

Not sure why the member profiles are included, but they should not if guests do not have permissions. I'll ask Oleg to take a look.

theguru · #5 04-22-2007, 08:12 PM

Sorry, but I have to disagree here. There are very specific reasons why we require members to be logged in to view thread contents. The use of giving spiders specific permissions other than guest permissions has also been approved by Google (in fact we have actually took part in a trial where we provided Google with login details especially for spidering).

Now I have also just noticed a discrepency with permissions. When I ran the Scheduled Task manually, it actually did almost what I am looking for and produced

Quote:

Summary:
============================
Forum Display: 826
Show Thread: 47284
Show Post: 346872
Member Profiles: 35882
Poll Results: 1802
Archive: 261

Total Indexed URLs: 432928
Total Processing Time: 56.41 seconds

Google ping: Successful.
Yahoo ping: Successful.

I figure this is because it used my permissions at the time it was run?

Keith Cohen · #6 04-22-2007, 08:27 PM

If you run it manually through the AdminCP, it will run under your permissions, and even include threads in private Admin/Moderator forums you may have. If you run it manually from the vBSEO Sitemap menu, it will run as normal.

theguru · #7 04-22-2007, 08:31 PM

If I run it manually through the AdminCP, will it still exclude the forums I have listed to be excluded?

Keith Cohen · #8 04-22-2007, 08:33 PM

Yes, it should.

Mike · #9 04-22-2007, 08:53 PM

Quote:

Originally Posted by Keith Cohen

If you run it manually through the AdminCP, it will run under your permissions, and even include threads in private Admin/Moderator forums you may have.

Well, that is good to know. I had always wondered why my totals were higher when running through the admin control panel. Won't be doing that anymore.

theguru · #10 04-22-2007, 08:54 PM

Excellent. So for now I will just have to run generate the sitemap manually in order to get it to do what I need it to. Please don't change that

Thanks for your help Keith.

theguru · #11 04-22-2007, 08:55 PM

Quote:

Originally Posted by Mike

Well, that is good to know. I had always wondered why my totals were higher when running through the admin control panel. Won't be doing that anymore.

Just exclude your staff folders Mike. The piece of mind is worth the little effort

Mike · #12 04-22-2007, 09:41 PM

Quote:

Originally Posted by theguru

Just exclude your staff folders Mike. The piece of mind is worth the little effort

Right, I just assumed that guest permissions were being followed from the admincp. Will definitely exclude them just in case I use the admincp to submit in the future.

Keith Cohen · #13 04-22-2007, 10:05 PM

Worst case, someone who went in and downloaded your sitemaps would see thread titles from private forums. They wouldn't see any content.

Ace Shattock · #14 04-22-2007, 10:43 PM

So, it would only really be a problem if you discussed things with relevant thread titles..

http://domain.com/forum/admin-stuff/...d-kicking.html

theguru · #15 04-23-2007, 08:06 AM

Quote:

Originally Posted by Keith Cohen

Worst case, someone who went in and downloaded your sitemaps would see thread titles from private forums. They wouldn't see any content.

This is actually why I would like to see the sitemap generated to include threads even though a user must be logged in to view the content. It generates interest because potential members can see the subject of what is being discussed, but if they want to read the guts of it ... the least they can do is register and sign in.