Detailed break down of events:

1. We get slashdotted, SF.net newslettered and are serving 400k banners a day of our web site
2. Through a perfect alignment of heavy traffic, we see heavy & poorly handled load on the web heads.  Here's the CPU usage, load averge and swap usage leading up to the crash:

		CPU USAGE          %user     %nice   %system   %iowait     %idle
	07:40:02 AM           31.38      0.00      6.12     10.32     52.17
	07:50:48 AM           30.01      0.00     12.39     35.63     21.97
	08:13:19 AM           2.22      0.00      8.06     89.72      0.00

		LOAD AVERAGE  runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15
	07:40:02 AM         2       204     12.76      5.89      2.76
	07:50:48 AM         2       353    162.85     83.22     34.71
	08:13:19 AM         0       394    253.03    247.96    189.54

		SWAP USAGE	pswpin/s pswpout/s
	07:40:02 AM    208.47    394.31
	07:50:48 AM    460.32    973.28
	08:13:19 AM    398.44    372.77

3. Each web head was set to have 128 apache children.  In our testing on staging, we saw the children swell from 24-30megs each.  That's bad math for only 1GB of RAM.  128 children * 25 megs (at best)/child = 3,200 megs of RAM needed!! That means swap, which lines up with the sar results we saw above. as well as the seg faults (see below)

4.  With potentially 256 apache children making a DB request, mysql started turning people away as it was set to 100 max connections (128 children * 2 web heads = 256) . The DB server, interestingly, was not working hard when the web heads were.  This means we can probably up the number or connections later:

		CPU USAGE          %user     %nice   %system   %iowait     %idle
	07:50:01 AM            6.17      0.00      5.36      0.52     87.95
	08:00:01 AM            2.89      0.00     24.27      0.07     72.77
	08:10:01 AM            7.57      0.00     17.76      0.02     74.64

	LOAD AVERAGE  runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15
	07:50:01 AM         1       209      0.49      0.42      0.84
	08:00:01 AM         4       217      1.38      1.09      0.94
	08:10:01 AM         1       213      0.52      0.83      0.91
	08:20:01 AM         1       161      0.62      0.75      0.80

	SWAP USAGE	pswpin/s pswpout/s
	07:50:01 AM      0.00      0.00
	08:00:01 AM      0.00      0.00
	08:10:01 AM      0.00      0.00
	08:20:01 AM      0.00      0.00

5. The web  heads swapping like crazy, apache children segfaulting, CPU usage v. high, get forcefully rebooted by sean and reset the load.

The answer then is to set the number of apache children to aprox 40/web head.  This will ensure that we don't run out of the RAM we have now (1gb) as well as not exceed the max connections to mysql (40 + 40 != 100).

-adj