High CPU Loads: AMD64, RedHat

I am experiencing incredibly high CPU loads that crash my server several times a day and have been difficult to track down and resolve. The machine can still be pinged, but must be hard booted in order to restore access to vBulletin. The forum usually has about 200 simultaneous users and 2.9 million posts.

The specs are: Dual AMD Opteron 265, 4GB RAM, (4) 73GB SCSI drives in RAID 5, RedHat ES 4 Update 4 (2.6.9-42.0.3.ELsmp), Apache 2.2.3, PHP 4.4.4. Running Zend Optimizer and eaccelerator 0.9.5. MySQL runs on a separate server which does not show any load problems, so this is all httpd based.

Code:

top - 18:10:44 up  1:14,  1 user,  load average: 107.98, 63.23, 61.83
Tasks: 207 total,  26 running, 181 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.6% us, 98.3% sy,  0.0% ni,  0.0% id,  0.0% wa,  0.0% hi,  0.1% si
Mem:   4037924k total,  4022208k used,    15716k free,      364k buffers
Swap:  2031608k total,  2031608k used,        0k free,    15000k cached
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
 5128 apache    16   0  405m 286m 5892 D   13  7.3   0:16.92 httpd              
   76 root      16   0     0    0    0 S   11  0.0   1:30.50 kswapd1            
 5010 apache    16   0  154m  31m 6940 R   10  0.8   0:11.43 httpd              
 5117 apache    16   0  145m  30m 6556 R    9  0.8   0:09.27 httpd              
 5140 apache    16   0  180m  65m 5852 D    9  1.7   0:07.10 httpd              
 5079 apache    16   0  170m  51m 6172 R    8  1.3   0:11.37 httpd              
 5070 apache    16   0  121m 3292 1768 D    8  0.1   0:00.91 httpd              
 5023 apache    16   0  183m  48m 7040 R    8  1.2   0:21.97 httpd              
 5076 apache    15   0  224m  69m 6496 D    8  1.8   0:12.30 httpd              
 5057 apache    16   0  224m  60m 7284 D    8  1.5   0:18.02 httpd              
   77 root      16   0     0    0    0 S    7  0.0   2:05.67 kswapd0            
 5051 apache    15   0  575m 202m 7008 D    7  5.1   0:20.87 httpd              
 5092 apache    16   0  452m 168m 6348 R    7  4.3   0:14.89 httpd              
 5082 apache    15   0  145m  26m 6444 D    7  0.7   0:06.41 httpd              
 5091 apache    16   0  152m  24m 6880 R    7  0.6   0:06.93 httpd              
 5093 apache    15   0  147m  23m 7152 D    7  0.6   0:08.83 httpd              
 5155 apache    15   0  125m  11m 4736 D    7  0.3   0:02.24 httpd

Any help would be much appreciated.

Hello DaVe,

could you please open a support ticket with ftp/admincp access for troubleshooting?

Just to let everyone know, I finally fixed the problem. I'm not sure which one of these steps did it, but I:

1. Upgraded from PHP 4.4.4 to PHP 5.1.6
2. Replaced the php.ini file with the default and then edited it line by line
3. Recompiled and reinstalled eaccelerator 0.9.5
4. Reinstalled Zend optimizer 3.0.2

Been up for almost 24 hours now with a load average of under 2, which is great.

Great, thank you for the follow-up!