SEO sitemap generation fails

When I run the sitemap generation manually from "vBSEO Sitemap Reports", I get the output below. It then hangs (even after waiting 2 hours). I have repeated this a few times and got the same problem. I got the same result by running the cron (sched task) manually via admin CP.

Admin Control Panel (vBulletin 4.0.3 Patch Level 1)
[homepage] [10,360.7Kb mem used] [0s (+0s)]
[SECTION START] forumdisplay [10,363.6Kb mem used] [0s (+0s)]
[forumdisplay] forum_id: 41, total threads: 0, pages: 1 [10,369.2Kb mem used] [0s (+0s)]
[forumdisplay] forum_id: 50, total threads: 862, pages: 44 [10,718.3Kb mem used] [0s (+0s)]
[forumdisplay] forum_id: 42, total threads: 726, pages: 37 [10,768.8Kb mem used] [0s (+0s)]
[forumdisplay] forum_id: 49, total threads: 0, pages: 1 [10,811.3Kb mem used] [0s (+0s)]
[forumdisplay] forum_id: 1, total threads: 0, pages: 1 [10,812.6Kb mem used] [0s (+0s)]
[forumdisplay] forum_id: 19, total threads: 200, pages: 10 [10,813.8Kb mem used] [0s (+0s)]
[forumdisplay] forum_id: 20, total threads: 361, pages: 19 [10,825.0Kb mem used] [0s (+0s)]
[forumdisplay] forum_id: 4, total threads: 0, pages: 1 [10,846.4Kb mem used] [0s (+0s)]
[forumdisplay] forum_id: 7, total threads: 6154, pages: 308 [10,847.7Kb mem used] [0s (+0s)]
[forumdisplay] forum_id: 5, total threads: 0, pages: 1 [11,197.8Kb mem used] [0s (+0s)]
[forumdisplay] forum_id: 45, total threads: 1193, pages: 60 [11,199.1Kb mem used] [0s (+0s)]
[forumdisplay] forum_id: 44, total threads: 1, pages: 1 [11,265.3Kb mem used] [0s (+0s)]
[forumdisplay] forum_id: 10, total threads: 3443, pages: 173 [11,266.6Kb mem used] [0s (+0s)]
[forumdisplay] forum_id: 26, total threads: 563, pages: 29 [11,466.6Kb mem used] [0s (+0s)]
[forumdisplay] forum_id: 25, total threads: 1217, pages: 61 [11,498.9Kb mem used] [0s (+0s)]
[forumdisplay] forum_id: 38, total threads: 217, pages: 11 [11,566.8Kb mem used] [0s (+0s)]
[forumdisplay] forum_id: 34, total threads: 131, pages: 7 [11,579.1Kb mem used] [0s (+0s)]
[forumdisplay] forum_id: 48, total threads: 470, pages: 24 [11,587.0Kb mem used] [0s (+0s)]
[forumdisplay] forum_id: 12, total threads: 4665, pages: 234 [11,614.1Kb mem used] [0s (+0s)]
[forumdisplay] forum_id: 21, total threads: 355, pages: 18 [10,912.4Kb mem used] [0s (+0s)]
[forumdisplay] forum_id: 47, total threads: 165, pages: 9 [10,940.6Kb mem used] [0s (+0s)]
[forumdisplay] forum_id: 55, total threads: 10, pages: 1 [10,950.8Kb mem used] [0s (+0s)]
[forumdisplay] forum_id: 37, total threads: 478, pages: 24 [10,952.1Kb mem used] [0s (+0s)]
[forumdisplay] forum_id: 24, total threads: 773, pages: 39 [10,979.4Kb mem used] [0s (+0s)]
[forumdisplay] forum_id: 6, total threads: 0, pages: 1 [11,022.9Kb mem used] [0s (+0s)]
[forumdisplay] forum_id: 13, total threads: 4709, pages: 236 [11,024.1Kb mem used] [0s (+0s)]
[forumdisplay] forum_id: 40, total threads: 0, pages: 1 [11,287.8Kb mem used] [0s (+0s)]
[forumdisplay] forum_id: 23, total threads: 17, pages: 1 [11,289.0Kb mem used] [0s (+0s)]
[forumdisplay] forum_id: 22, total threads: 393, pages: 20 [11,290.3Kb mem used] [0s (+0s)]
[forumdisplay] forum_id: 39, total threads: 32, pages: 2 [11,312.6Kb mem used] [0s (+0s)]
[forumdisplay] forum_id: 46, total threads: 48, pages: 3 [11,315.0Kb mem used] [0s (+0s)]
[forumdisplay] forum_id: 51, total threads: 0, pages: 1 [11,318.5Kb mem used] [0s (+0s)]
[forumdisplay] forum_id: 54, total threads: 0, pages: 1 [11,319.8Kb mem used] [0s (+0s)]
[forumdisplay] forum_id: 53, total threads: 1, pages: 1 [11,321.1Kb mem used] [0s (+0s)]
[forumdisplay] forum_id: 52, total threads: 0, pages: 1 [11,322.3Kb mem used] [0s (+0s)]
[forumdisplay] forum_id: 56, total threads: 0, pages: 1 [11,323.6Kb mem used] [0s (+0s)]
[SECTION START] showthread [11,318.2Kb mem used] [0s (+0s)]

When I look on the filesys on the server, I get this:
[root@myprotein data]# ls -lrt
total 300
-rwxrwxrwx 1 community psacln 0 Feb 8 2011 index.php
drwxrwxrwx 2 community psacln 4096 Sep 23 11:35 hits
-rw-r--r-- 1 apache apache 6 Sep 23 14:18 downloads.dat
-rw-r--r-- 1 apache apache 197917 Sep 23 14:18 sitemap_forum_1.xml.gz
-rw-rw-rw- 1 apache apache 88221 Sep 23 14:18 progress.dat


So it has actually generated one sitemap file, sitemap_forum_1.xml.gz. The file has 1000 urls in it (I have a limit of 30K set in admin CP. Further, the sitemap_forum_1.xml.gz is not actually gzipped, as I can 'more' it and see the xml sitemap file contents.

Any ideas as to why the sitemap gen process seems to stop or hang?

Thanks

Andy

Hello,

It's likely timing out.

You may need to increase your wait_timeout setting in my.cnf (try 180) and/or increase your php.ini memory_limit setting (try 64M)

Thank you for your reply Brian.

I first added a new entry to /etc/my.conf “wait_timeout=180” (there was no other entry for wait_timeout). I then restarted mysqld, cleaned up the .dat files and .xml.gz file in vbseo_sitemap/data.

I then re-ran the sitemap generation. It failed again. Again, there was only 1 sitemap file and it contained 1000 urls (just as before). Seems a strange nice round number.

I then went to /etc/php.ini. memory_limit was already set to 256M so I left that alone. I changed max_execution_time from 30 to 600.

max_execution_time = 600 ; Maximum execution time of each script, in seconds
max_input_time = 60 ; Maximum amount of time each script may spend parsing request data
memory_limit = 256M

I restarted httpd, cleared up the sitemap and .dat files as above and re-ran the sitemap gen.

Same result as before.

[root@myprotein data]# grep '<loc>' sitemap_forum_1.xml.gz | wc -l
1000

Is there a limit of for the 1000 set somewhere in a param?

The file count limit is set directly from vboptions -> sitemap generator options in the 2nd option for how many urls per file. 20,000 is a general recommended number.

If it's stopping at the same item every time, it's possible it's throwing an error_log entry. Run it and check that again. You may have some corrupt data/etc that is causing it to fail out every time.

Brian

Thanks again for the reply.

I have 30000 urls per file limit set (this should be fine I believe). I looked at the error_log and I think that this identifies the issue:

[Wed Sep 28 14:38:07 2011] [error] [client 81.102.XX.YY] PHP Fatal error: Call to undefined method vB_Friendly_Url_Error::get_url() in /var/www/vhosts/community.XXX.com/httpdocs/includes/functions.php on line 6876, referer: http://XXX.myprotein.com/vbseo_sitem...php?rlist=true

I assume that this is part of the SEO url re-writer for friendly urls? Friendly urls appear to work fine on the actual site. Any idea how to resolve this error?

Andy

I found this: Fatal error: Call to undefined method vB_Friendly_Url_Error

I too disabled the 'adding showpost links to sitemaps' option and the issue 'went away'. I would welcome any advise on wheather I should actuallty have showpost links in the sitemap.

Thanks

Andy

You can also try to re-import the vbseo sitemap product xml. It should over-ride any bad settings from the vb friendly urls. Also make sure the option for 'vbseo installed' is correctly set to yes.