Site Map is not listing my non-forum URLs

[eJM]

I checked my sitemap url list tonight and notice, to my dismay, that it contained no URLs from the rest of my site. I did a search for issues related to this and found that it was recommended that a line of code should be added to the .htaccess file that is in the domain root. I checked my .htaccess file and there is a line of code there, just not exactly the same as what I found in an older message from 2005.

Here is what was recommended in another discussion from 2005:

RewriteRule ^(sitemap.*\.(xml|txt)(\.gz)?)$ /forums/vbseo_sitemap/vbseo_getsitemap.php?sitemap=$1 [L]

Here is what I have in my own file:

RewriteRule ^((urllist|sitemap).*\.(xml|txt)(\.gz)?)$ community/vbseo_sitemap/vbseo_getsitemap.php?sitemap=$1 [L]

I know, of course, that the directory "forums" in the top code would need to be changed to whatever I use as the directory for my forums, which is what I have in my own file, as you can see from the second code.

The problem is getting all the URLs that are NOT in the forums listed and that's not happening. The indication was that the line of code would fool the generator into looking at the whole site, not just the forums. On top of all this, I also have the add-on for vBadvanced in the proper location, which is the program I use to make all of the rest of my pages.

Can someone tell me how I can edit my .htaccess file to work properly, or help me edit the appropriate file(s) that will allow all of my pages (except private ones, of course) to be listed in the site map? Thanks for any assistance on this.

Jim

[Oleg Ignatiuk]

Hello Jim,

vBSEO Sitemap Generator automatically includes URLs for forum only, it is not designed to generate sitemap for non-vBulletin pages.
It does allow though to create your custom add-on (like addons/vbseo_sm_vba.php example) OR manually add all non-vB urls in extra-urls.txt file (one per line).

Rewrite rule in domain root .htaccess allows to access sitemap via top domain URL, but it doesn't affect sitemap files and doesn't include any additional URLs.

[eJM]

Originally Posted by Oleg Ignatiuk

vBSEO Sitemap Generator automatically includes URLs for forum only, it is not designed to generate sitemap for non-vBulletin pages.
It does allow though to create your custom add-on (like addons/vbseo_sm_vba.php example) OR manually add all non-vBulletin urls in extra-urls.txt file (one per line).

As I mentioned in my post above, I do have the vBa add-on:

Originally Posted by Me

On top of all this, I also have the add-on for vBadvanced in the proper location, which is the program I use to make all of the rest of my pages.

So, I am wondering why it's not including URLs from all my vBa pages? Am I supposed to edit the vbseo_sm_vba.php file or something? This is an add-on that was included in the sitemap script and should have had instructions for use included. Likewise the extra-urls.txt file -- there were absolutely no instructions on how to use this empty file. You instructions above were the first that actually said what the format of that file should be (according to my search yesterday).

Is noppid reading this?: can you give us an idea of how you edited your sitemap to include your non-forum pages? Thanks.

Rewrite rule in domain root .htaccess allows to access sitemap via top domain URL, but it doesn't affect sitemap files and doesn't include any additional URLs.

Okay, so I misunderstood the purpose of putting that line of code in my domain level .htaccess file. It really serves no viable function, it looks like. It takes extra processing power and does nothing that having the code in the forum level .htaccess file doesn't do. It is apparently not necessary. Is that correct? I don't want to waste cycles or increase the size of my files if it doesn't serve an important function.

I think part of the problem may be that there is so little information available to the novice concerning this script. The instructions included are seriously lacking. I know that it's a beta and I have heard the excuse about a lack of instructions with beta versions, but honestly, we're taking this beta thing a bit too far. Everything here is either a beta or RC version. Let's get on with the real deal and include some useful documentation with it.

Thanks for understanding my frustration and for any assistance with these issues.

Jim

[eJM]

I hope it's not because of a little criticism that there has been no reply yet.

I opened the addons/vbseo_sm_vba.php file and edited it. I removed the comment, as directed in the file (instructions should have been provided to tell us that the file could or should be edited), to indicate that vBa was located in an upper level directory:

<?php
	define(VBSEO_NOTLOGGEDIN_GID, 1); // change this to your "Unregistered / Not Logged In Usergroup ID"
	$vba_url = $vbseo_vars['bburl'] . '/';

	// uncomment the line below if your vbadvanced root URL is in upper level relative to vBulletin url
	$vba_url = preg_replace('#[^/]*/$#', '', $vba_url);

	$vba_options = array();

	$q_opts = $db->query_read("SELECT * FROM " . TABLE_PREFIX . "adv_setting");
	while ($opt = $db->fetch_array($q_opts))
		$vba_options[$opt['varname']] = $opt['value'];

	$mods = $db->query_read("SELECT * FROM " . TABLE_PREFIX . "adv_pages WHERE active = 1 ORDER BY displayorder");
	while ($mod = $db->fetch_array($mods))
	if(in_array(VBSEO_NOTLOGGEDIN_GID, explode(',',$mod['userperms'])))
	{	
		$url = $vba_options['portal_homeurl'].'?'.$vba_options['portal_pagevar'].'='.$mod['name'];

		if(VBSEO_ON)
			$url = vbseo_any_url($url);

		if(!strstr($url, '://'))
			$url = $vba_url.$url;

  		vbseo_add_url($url, 1.0, '', 'daily');
	}

?>

The sitemap was generated tonight and it included 21 new and non-forum pages. A step in the right direction. Unfortunately, the URLs reported are not the URLs I use on my site. All the pages I build using vBa are designed to be static URLs. Instead, the sitemap reports them all as dynamic URLs. Example:

Report - http://www.thefloorpro.com//index.php?page=home

is actually - http://www.thefloorpro.com/index.php

Report - http://www.thefloorpro.com//index.php?page=about

is actually - http://www.thefloorpro.com/about.php

How can I correct this?

How can I prevent certain pages from being reported?

Should I open a support ticket for this, or can staff help me with this on the forum so that anyone using vBa to build static pages will have the benefit of this information too?

Thank you,

Jim

[Keith Cohen]

The CMPS add-on module is provided as an example for people who wish to create their own modules, and is not "officially" supported. As you found, the code contains comments to guide you.

If you use CRR's to rewrite your vBadvanced URLs, and you want those to be the URLs included in your sitemap, then you will need to manually put those into your extra-urls.txt file. Otherwise, leave it as-is and Google will be 301 redirected to the rewritten URLs.

[eJM]

Originally Posted by Keith Cohen

The CMPS add-on module is provided as an example for people who wish to create their own modules, and is not "officially" supported. As you found, the code contains comments to guide you.

Wait a minute here. You (Crawlability) provide a file that the user is directed to upload along with all the other files and folders. Then when a user finds it isn't working correctly, either because it's not coded right or because the instructions for use were left out, you tell us it's not "officially" supported.

Nowhere in the documentation does it say this is just an example and we are on our own to try and figure it out. The documentation leaves out other important stuff too, like about the includes folder and how to install on a 3.6.x system and that this is just a beta version, etc. When are you going to take responsibility for the proper assembly, distribution and support of your product?

This is very disturbing. I almost feel like I have been bamboozled. You put out an excellent product in vBSEO (I won't go into the ethical and developmental issues with that) and enhance its value with products like the vBSEO Sitemap Generator, but continually falter in the promotions and documentations departments. Maybe it's time for Crawlability to collectively get their shxx together.

If you use CRR's to rewrite your vBadvanced URLs, and you want those to be the URLs included in your sitemap, then you will need to manually put those into your extra-urls.txt file. Otherwise, leave it as-is and Google will be 301 redirected to the rewritten URLs.

Leaving it as-is did not redirect me to the rewritten URLs, so how would it have redirected Google? It also doesn't appear to have worked to have the line of code in my domain root .htaccess file. The location of my sitemap is always indicated as in my forums folder. I asked about this code in my original message to this topic, but was not told which line of code that I posted was the correct one. I'd also like a straight answer as to whether having the code in my domain root .htaccess file will have any advantage at all or whether it is just taking up space.

Thank you for addressing these issues, if you will.

Jim

[Keith Cohen]

Wait a minute here. You (Crawlability) provide a file that the user is directed to upload along with all the other files and folders. Then when a user finds it isn't working correctly, either because it's not coded right or because the instructions for use were left out, you tell us it's not "officially" supported.

As I said, it's included as an example. It isn't even used unless you add it to the vBSEO Sitemap AdminCP section telling it to use that module. Consider it a SDK (Software Development Kit). If you aren't comfortable developing, you should probably post a request and see if any other users have developed any modules that will work for your needs.

Nowhere in the documentation does it say this is just an example and we are on our own to try and figure it out.

Where in the documentation does it tell you to use it?

Leaving it as-is did not redirect me to the rewritten URLs, so how would it have redirected Google?

If going to your CMPS dynamic links doesn't redirect you to the static links, then something isn't right with your CRRs or with the rewrite rules in your root .htaccess file. You can seek help in the CRR (Custom Rewrite Rules) forum here on vBSEO.com.

I asked about this code in my original message to this topic, but was not told which line of code that I posted was the correct one. I'd also like a straight answer as to whether having the code in my domain root .htaccess file will have any advantage at all or whether it is just taking up space.

If all URLs in your sitemap are urls that are under your forums directory, then you don't need any extra lines in your root .htaccess, since you won't be submitting sitemaps from that location. If you ARE including URLs from the root of your site (via extra-urls.txt or a module), then you will want to have that line in your root .htaccess file, and submit your sitemaps using your root URL. (You will also want to change the vBSEO Sitemap AdminCP option that lets you specify the location of your sitemaps. It defaults to blank. If you are submitting from your root URL, put your root URL in that field.)

Use the same sitemap line that is included in the .htaccess that comes with vBSEO. Copy that one line (along with RewriteEngine On if you don't already have it) and put that in your root .htaccess.

[Oleg Ignatiuk]

All the pages I build using vBa are designed to be static URLs. Instead, the sitemap reports them all as dynamic URLs. Example:

What is your vBa CMPS homepage URL as defined in admincp?

[eJM]

Originally Posted by Oleg Ignatiuk

What is your vBa CMPS homepage URL as defined in admincp?

Thank you for your reply, Oleg. The home page URL is /index.php, which translates to http://www.thefloorpro.com/index.php. I also have my site organized into other directories, such as http://www.thefloorpro.com/reviews/index.php, http://www.thefloorpro.com/articles/index.php and http://www.thefloorpro.com/how-to/index.php, just for examples. Each of those directories also have static pages, such as, http://www.thefloorpro.com/reviews/ultimate_scraper_blades_review.php. And there are static pages off the root too, such as, http://www.thefloorpro.com/reviews/about.php.

R'gards,

Jim

[Oleg Ignatiuk]

You should change your CMPS homepage link to the full URL :

http://www.thefloorpro.com/index.php

[eJM]

Originally Posted by Keith Cohen

As I said, it's [vbseo_sm_vba.php] included as an example. It isn't even used unless you add it to the vBSEO Sitemap AdminCP section telling it to use that module. Consider it a SDK (Software Development Kit). If you aren't comfortable developing, you should probably post a request and see if any other users have developed any modules that will work for your needs.

It is included in the package and options for its use are in the sitemap configuration control panel. NO instructions for its use are included, but the options seem simple enough. My indication is that it is part and parcel of the sitemap product. Why would I think any different? Why would my use of the product be considered SDK use, especially since there is no indication of that anywhere in the CP or documentation? Why would I think to ask if a vBa module has been developed when there is already one, put there by Crawlability, in the package I downloaded?

Where in the documentation does it tell you to use it?

It's included in the package. So is the includes folder and the file in it. Does your reasoning apply to that as well? Are you telling me I should make assumptions for one, but not the other? Is the use of the sitemap supposed to be a guessing game?

If going to your CMPS dynamic links doesn't redirect you to the static links, then something isn't right with your CRRs or with the rewrite rules in your root .htaccess file. You can seek help in the CRR (Custom Rewrite Rules) forum here on vBSEO.com.

I will check into that further.

If all URLs in your sitemap are urls that are under your forums directory, then you don't need any extra lines in your root .htaccess, since you won't be submitting sitemaps from that location. If you ARE including URLs from the root of your site (via extra-urls.txt or a module), then you will want to have that line in your root .htaccess file, and submit your sitemaps using your root URL. (You will also want to change the vBSEO Sitemap AdminCP option that lets you specify the location of your sitemaps. It defaults to blank. If you are submitting from your root URL, put your root URL in that field.)

As I hope I have made clear here, I am including URLs from the root of my domain. I added my root URL to the CP as you said. I changed the CMPS home page URL as Oleg suggested in his last message. When I try to download the compressed text list and the other compressed XML files, they can't be found. This seems to be getting worse, not better.

Use the same sitemap line that is included in the .htaccess that comes with vBSEO. Copy that one line (along with RewriteEngine On if you don't already have it) and put that in your root .htaccess.

The one that comes with vBSEO is the same one I have in my root's .htaccess file, EXCEPT for the inclusion of community/ in my root .htaccess file. If I am not mistaken, Oleg put that line there on a previous support call into my site. I tried running it after changing the line to take out community/, but it still doesn't work. I cannot download any of the sitemap files, which probably means Google and Yahoo can't either.

I hope we can get this worked out.

Jim

[Oleg Ignatiuk]

Please open a support ticket with ftp/vbseo_sitemap access for troubleshooting.

[eJM]

Will do, Oleg, thank you.

Jim

[goodgirl]

Originally Posted by Oleg Ignatiuk

Hello Jim,

vBSEO Sitemap Generator automatically includes URLs for forum only, it is not designed to generate sitemap for non-vBulletin pages.
It does allow though to create your custom add-on (like addons/vbseo_sm_vba.php example) OR manually add all non-vBulletin urls in extra-urls.txt file (one per line).

Rewrite rule in domain root .htaccess allows to access sitemap via top domain URL, but it doesn't affect sitemap files and doesn't include any additional URLs.

Is there any plans to make it easier to add static pages and choose where the location of the sitemap will be? Something like Google Sitemap Generator for WordPress v2 Final did for wordpress would be extremely helpful. Would it be that hard for VbSEO to allow us to choose a manual location for the sitemap and have an easy interface to add static pages?

It really doesn't seem like a bad request.

[Oleg Ignatiuk]

Hello Tanya,

There is a special option in vBSEO Sitemap Generator options that should be used if you want to "move" sitemap to a non-vbulletin folder: "Sitemap Files URL". .htaccess file should be created in a target folder as well, as described above.

Adding more static pages into sitemap is as easy as inserting the URLs into extra-urls.txt file