20X Peformance Drop with VBSEO

I've sent a ticket to support but am posting here in case the community has seen a similar problem.

Server details listed below but the meaningful comparison is with vs. without VBSEO on the same hardware.

Without VBSEO the site is very fast with loads around 1. With VBSEO, the server grinds to a halt unless I use VB's throttling.

Without VBSEO, load average hovers around 1 on the web server and below 1 on the database server. There is little swap usage and no disk io issues. CPU is 74% idle on the Web and 65% idle on the DB. At peak, loads on the web never go above 2.

With VBSEO enabled, loads surge to 20+ until the VB throttling kicks in. Loads surge even higher without the throttling. Memory usage changes little but nearly all of the Web CPU is used with VBSEO enabled. The DB CPU/RAM metrics are largely unchanged.

So I am wondering why VBSEO is so CPU intensive? With VBSEO enabled, I notice little load increase on the database server, so I don't think the DB is causing the issue unless there is a lock issue.

Something either in the rewrite phase (maybe I've the wrong rules. Using the defaults in the download kit) or VBSEO itself is problematic.

Any insights appreciated.

Server Details:
CentOS 5 64 Bit SMP with default Apache (2.2) and PHP (5.16) and MySQL 5. Zend 3.3.

VB is using memcache for the data-store and PHP has eAccelerator. (Disabling these did not make any difference).

Using connects but pconnects did not help. No difference between mysql/mysqli.

Servers are Dual Woodcrest 5130's with RAID 1+0. The DB has 8GB RAM and 15K SAS while the Web has 4GB RAM and 10K RPM SATA.

Web connects to the DB over a private GigE link.

One thing I want to do is look at the Web to DB traffic with and without VBSEO. Perhaps the DB server is handling it fine but the web server cannot cope with all of the extra processing VBSEO requires.

The site's had performance issues for a while. We've updated servers and standardized on Linux. The issue is not with the new servers. There are no performance issues if VBSEO is disabled and the rewrites removed. If I enable the module performance is terrible.

If I made only this change:
Goto the VBSEO control panel, disable module and remove rewrite rules, the loads drop from a throttled level of 20 to near 1. This is easily reproducible and directly related to using the VBSEO module.

I have got quitte some time troubles with vbSEO performance. I fixed it by upgrading to php 5.2.3 with the ioncube version, and installing xcache. THis has improved performance greatly. Maybe this can help you to?

We are on Red Hat's PHP -- may custom build later if needed.

We switched to lighttpd + fast-cgi + xcache and it is working well. Will try ioncube version to see if that is better.

Which version is better for performance, Zend or Ioncube???

We had better performance with ioncube.

FYI: I am posting for Ismet. JH at Linux Server Management Security Backup and Support Solutions by rackAID

We are still having performance issues with the system. I am not sure there is much more we can eek out of the system because the performance of VBSEO is simply poor compared to running without it.

We are trying to track down where the added CPU load happens. I am not a php developer but a sysadmin, so I've no idea what you guys are doing in the code --- nor could we know since it is closed source.

One issue may be the re-write rules. I am not sure they sufficiently trap out all requests that are not necessary to re-write. We will have to check that logic more carefully.

We are just trying to eek out any performance benefit we can by tuning the configuration before we start stacking up more servers.

Are there any reasons for POSTs to be re-written? I am wondering if there may be so much load created because many pages get passed into vbseo that may not need to be?

We've made some progress by adjusting the re-write rules for lighttpd. The exclusions should go to $0 not $1, this avoids the internal re-write process completely.

Also, we keep adding as many exclusions to rewrite rules as possible to limit any re-writing.

We did have to add another web server (dual quad core xeon) and will see how peak hours go tomorrow.

I cannot stress how much memcache helps with DB loads. If you are having VB performance issues, adding memcache is highly recommended.

Right now we are running isolated memcache instances, but plan to test these versus the pooled instances.

Also, we are looking at moving to a master/multi-slave setup. We will run combination www/msyql slave setups so that the VB forum can do reads via localhost and use a socket instead of a tcp connection.

Jeff from Linux Server Management Security Backup and Support Solutions by rackAID