Apr 22 06:23:00 box9 kernel: mysqld invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0 Apr 22 10:53:54 box9 clamd[2339]: SelfCheck: Database status OK. Apr 22 10:54:26 box9 kernel: Apr 22 10:54:53 box9 no crontab for spoloh Apr 22 10:54:52 box9 kernel: Call Trace: Apr 22 10:54:54 box9 kernel: [] out_of_memory+0x8e/0x2f3 Apr 22 10:54:54 box9 kernel: [] __alloc_pages+0x27f/0x308 Apr 22 10:54:54 box9 kernel: [] :dm_mod:dm_table_any_congested+0x53/0xa3 Apr 22 10:54:54 box9 kernel: [] __do_page_cache_readahead+0x96/0x179 Apr 22 10:54:54 box9 kernel: [] filemap_nopage+0x14c/0x360 Apr 22 10:54:54 box9 kernel: [] __handle_mm_fault+0x1fb/0x1039 Apr 22 10:54:54 box9 kernel: [] math_state_restore+0x23/0x4c Apr 22 10:54:54 box9 kernel: [] error_exit+0x0/0x84 Apr 22 10:54:54 box9 kernel: [] do_page_fault+0x4cb/0x874 Apr 22 10:54:54 box9 kernel: [] sys_rt_sigreturn+0x283/0x356 Apr 22 10:54:54 box9 kernel: [] sys_rt_sigreturn+0x323/0x356 Apr 22 10:54:54 box9 kernel: [] error_exit+0x0/0x84 Apr 22 10:54:54 box9 kernel: Apr 22 10:54:54 box9 kernel: Mem-info: Apr 22 10:54:54 box9 kernel: Node 0 DMA per-cpu: Apr 22 10:54:54 box9 kernel: cpu 0 hot: high 0, batch 1 used:0 Apr 22 10:54:54 box9 kernel: cpu 0 cold: high 0, batch 1 used:0 Apr 22 10:54:54 box9 kernel: cpu 1 hot: high 0, batch 1 used:0 Apr 22 10:54:55 box9 kernel: cpu 1 cold: high 0, batch 1 used:0 Apr 22 10:54:55 box9 kernel: cpu 2 hot: high 0, batch 1 used:0 Apr 22 10:54:55 box9 kernel: cpu 2 cold: high 0, batch 1 used:0 Apr 22 10:54:55 box9 kernel: cpu 3 hot: high 0, batch 1 used:0 Apr 22 10:54:55 box9 kernel: cpu 3 cold: high 0, batch 1 used:0 Apr 22 10:54:55 box9 kernel: Node 0 DMA32 per-cpu: Apr 22 10:54:55 box9 kernel: cpu 0 hot: high 186, batch 31 used:30 Apr 22 10:54:55 box9 kernel: cpu 0 cold: high 62, batch 15 used:20 Apr 22 10:54:55 box9 kernel: cpu 1 hot: high 186, batch 31 used:163 Apr 22 10:54:55 box9 kernel: cpu 1 cold: high 62, batch 15 used:57 Apr 22 10:54:55 box9 kernel: cpu 2 hot: high 186, batch 31 used:21 Apr 22 10:54:55 box9 kernel: cpu 2 cold: high 62, batch 15 used:51 Apr 22 10:54:55 box9 kernel: cpu 3 hot: high 186, batch 31 used:44 Apr 22 10:54:55 box9 kernel: cpu 3 cold: high 62, batch 15 used:49 Apr 22 10:54:55 box9 kernel: Node 0 Normal per-cpu: Apr 22 10:54:55 box9 kernel: cpu 0 hot: high 186, batch 31 used:161 Apr 22 10:54:55 box9 kernel: cpu 0 cold: high 62, batch 15 used:16 Apr 22 10:54:55 box9 kernel: cpu 1 hot: high 186, batch 31 used:90 Apr 22 10:54:55 box9 kernel: cpu 1 cold: high 62, batch 15 used:49 Apr 22 10:54:55 box9 kernel: cpu 2 hot: high 186, batch 31 used:30 Apr 22 10:54:55 box9 kernel: cpu 2 cold: high 62, batch 15 used:55 Apr 22 10:54:55 box9 kernel: cpu 3 hot: high 186, batch 31 used:12 Apr 22 10:54:55 box9 kernel: cpu 3 cold: high 62, batch 15 used:55 Apr 22 10:54:55 box9 kernel: Node 0 HighMem per-cpu: empty Apr 22 10:54:55 box9 kernel: Free pages: 40728kB (0kB HighMem) Apr 22 10:54:55 box9 kernel: Active:797521 inactive:964923 dirty:0 writeback:0 unstable:0 free:10182 slab:36738 mapped-file:135 mapped-anon:1771267 pagetables:142227 Apr 22 10:54:55 box9 kernel: Node 0 DMA free:10816kB min:12kB low:12kB high:16kB active:0kB inactive:0kB present:10404kB pages_scanned:0 all_unreclaimable? yes Apr 22 10:54:55 box9 kernel: lowmem_reserve[]: 0 3182 7852 7852 Apr 22 10:54:55 box9 kernel: Node 0 DMA32 free:23252kB min:4592kB low:5740kB high:6888kB active:1272500kB inactive:1566440kB present:3258984kB pages_scanned:4992956 all_unreclaimable? yes Apr 22 10:54:55 box9 kernel: lowmem_reserve[]: 0 0 4670 4670 Apr 22 10:54:55 box9 kernel: Node 0 Normal free:6660kB min:6740kB low:8424kB high:10108kB active:1917584kB inactive:2293252kB present:4782464kB pages_scanned:7032616 all_unreclaimable? yes Apr 22 10:54:55 box9 kernel: lowmem_reserve[]: 0 0 0 0 Apr 22 10:54:55 box9 kernel: Node 0 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Apr 22 10:54:55 box9 kernel: lowmem_reserve[]: 0 0 0 0 Apr 22 10:54:55 box9 kernel: Node 0 DMA: 4*4kB 6*8kB 4*16kB 4*32kB 3*64kB 3*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 2*4096kB = 10816kB Apr 22 10:54:55 box9 kernel: Node 0 DMA32: 15*4kB 15*8kB 94*16kB 6*32kB 12*64kB 3*128kB 3*256kB 2*512kB 0*1024kB 1*2048kB 4*4096kB = 23252kB Apr 22 10:54:55 box9 kernel: Node 0 Normal: 201*4kB 54*8kB 39*16kB 14*32kB 10*64kB 1*128kB 2*256kB 2*512kB 2*1024kB 0*2048kB 0*4096kB = 6660kB Apr 22 10:54:55 box9 kernel: Node 0 HighMem: empty Apr 22 10:54:55 box9 kernel: 264 pagecache pages Apr 22 10:54:55 box9 kernel: Swap cache: add 1159815, delete 1159814, find 203396/213215, race 0+46 Apr 22 10:54:55 box9 kernel: Free swap = 0kB Apr 22 10:54:55 box9 kernel: Total swap = 4192924kB Apr 22 10:54:55 box9 kernel: Free swap: 0kB Apr 22 10:54:55 box9 kernel: 2277376 pages of RAM Apr 22 10:54:55 box9 kernel: 284310 reserved pages Apr 22 10:54:55 box9 kernel: 28805831 pages shared Apr 22 10:54:55 box9 kernel: 1 pages swap cached Apr 22 10:54:55 box9 kernel: klogd invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0 Apr 22 10:54:55 box9 kernel: Apr 22 10:54:55 box9 kernel: Call Trace: Apr 22 10:54:55 box9 kernel: [] out_of_memory+0x8e/0x2f3 Apr 22 10:54:55 box9 kernel: [] __alloc_pages+0x27f/0x308 Apr 22 10:54:55 box9 kernel: [] :dm_mod:dm_table_any_congested+0x53/0xa3 Apr 22 10:54:55 box9 kernel: [] __do_page_cache_readahead+0x96/0x179 Apr 22 10:54:55 box9 kernel: [] filemap_nopage+0x14c/0x360 Apr 22 10:54:55 box9 kernel: [] __handle_mm_fault+0x1fb/0x1039 Apr 22 10:54:55 box9 kernel: [] do_page_fault+0x4cb/0x874 Apr 22 10:54:55 box9 kernel: [] do_syslog+0x186/0x3aa Apr 22 10:54:55 box9 kernel: [] autoremove_wake_function+0x0/0x2e Apr 22 10:54:55 box9 kernel: [] fget_light+0x31/0x7c Apr 22 10:54:55 box9 kernel: [] error_exit+0x0/0x84 Apr 22 10:54:55 box9 kernel: Apr 22 10:54:55 box9 kernel: Mem-info: Apr 22 10:54:55 box9 kernel: Node 0 DMA per-cpu: Apr 22 10:54:55 box9 kernel: cpu 0 hot: high 0, batch 1 used:0 Apr 22 10:54:55 box9 kernel: cpu 0 cold: high 0, batch 1 used:0 Apr 22 10:54:55 box9 kernel: cpu 1 hot: high 0, batch 1 used:0 Apr 22 10:54:55 box9 kernel: cpu 1 cold: high 0, batch 1 used:0 Apr 22 10:54:55 box9 kernel: cpu 2 hot: high 0, batch 1 used:0 Apr 22 10:54:55 box9 kernel: cpu 2 cold: high 0, batch 1 used:0 Apr 22 10:54:55 box9 kernel: cpu 3 hot: high 0, batch 1 used:0 Apr 22 10:54:55 box9 kernel: cpu 3 cold: high 0, batch 1 used:0 Apr 22 10:54:55 box9 kernel: Node 0 DMA32 per-cpu: Apr 22 10:54:55 box9 kernel: cpu 0 hot: high 186, batch 31 used:30 Apr 22 10:54:55 box9 kernel: cpu 0 cold: high 62, batch 15 used:20 Apr 22 10:54:55 box9 kernel: cpu 1 hot: high 186, batch 31 used:163 Apr 22 10:54:55 box9 kernel: cpu 1 cold: high 62, batch 15 used:57 Apr 22 10:54:55 box9 kernel: cpu 2 hot: high 186, batch 31 used:21 Apr 22 10:54:55 box9 kernel: cpu 2 cold: high 62, batch 15 used:51 Apr 22 10:54:55 box9 kernel: cpu 3 hot: high 186, batch 31 used:44 Apr 22 10:54:55 box9 kernel: cpu 3 cold: high 62, batch 15 used:49 Apr 22 10:54:55 box9 kernel: Node 0 Normal per-cpu: Apr 22 10:54:55 box9 kernel: cpu 0 hot: high 186, batch 31 used:161 Apr 22 10:54:55 box9 kernel: cpu 0 cold: high 62, batch 15 used:16 Apr 22 10:54:55 box9 kernel: cpu 1 hot: high 186, batch 31 used:90 Apr 22 10:54:55 box9 kernel: cpu 1 cold: high 62, batch 15 used:49 Apr 22 10:54:55 box9 kernel: cpu 2 hot: high 186, batch 31 used:30 Apr 22 10:54:55 box9 kernel: cpu 2 cold: high 62, batch 15 used:55 Apr 22 10:54:55 box9 kernel: cpu 3 hot: high 186, batch 31 used:12 Apr 22 10:54:55 box9 kernel: cpu 3 cold: high 62, batch 15 used:55 Apr 22 10:54:55 box9 kernel: Node 0 HighMem per-cpu: empty Apr 22 10:54:55 box9 kernel: Free pages: 40728kB (0kB HighMem) Apr 22 10:54:55 box9 kernel: Active:798029 inactive:964415 dirty:0 writeback:0 unstable:0 free:10182 slab:36738 mapped-file:135 mapped-anon:1771267 pagetables:142227 Apr 22 10:54:55 box9 kernel: Node 0 DMA free:10816kB min:12kB low:12kB high:16kB active:0kB inactive:0kB present:10404kB pages_scanned:0 all_unreclaimable? yes Apr 22 10:54:55 box9 kernel: lowmem_reserve[]: 0 3182 7852 7852 Apr 22 10:54:55 box9 kernel: Node 0 DMA32 free:23252kB min:4592kB low:5740kB high:6888kB active:1272212kB inactive:1566344kB present:3258984kB pages_scanned:5005670 all_unreclaimable? yes Apr 22 10:54:55 box9 kernel: lowmem_reserve[]: 0 0 4670 4670 Apr 22 10:54:55 box9 kernel: Node 0 Normal free:6660kB min:6740kB low:8424kB high:10108kB active:1919904kB inactive:2291316kB present:4782464kB pages_scanned:7056108 all_unreclaimable? yes Apr 22 10:54:55 box9 kernel: lowmem_reserve[]: 0 0 0 0 Apr 22 10:54:55 box9 kernel: Node 0 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Apr 22 10:54:55 box9 kernel: lowmem_reserve[]: 0 0 0 0 Apr 22 10:54:55 box9 kernel: Node 0 DMA: 4*4kB 6*8kB 4*16kB 4*32kB 3*64kB 3*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 2*4096kB = 10816kB Apr 22 10:54:55 box9 kernel: Node 0 DMA32: 15*4kB 15*8kB 94*16kB 6*32kB 12*64kB 3*128kB 3*256kB 2*512kB 0*1024kB 1*2048kB 4*4096kB = 23252kB Apr 22 10:54:55 box9 kernel: Node 0 Normal: 201*4kB 54*8kB 39*16kB 14*32kB 10*64kB 1*128kB 2*256kB 2*512kB 2*1024kB 0*2048kB 0*4096kB = 6660kB Apr 22 10:54:55 box9 kernel: Node 0 HighMem: empty Apr 22 10:54:55 box9 kernel: 264 pagecache pages Apr 22 10:54:55 box9 kernel: Swap cache: add 1159815, delete 1159814, find 203396/213215, race 0+46 Apr 22 10:54:55 box9 kernel: Free swap = 0kB Apr 22 10:54:55 box9 kernel: Total swap = 4192924kB Apr 22 10:54:55 box9 kernel: Free swap: 0kB Apr 22 10:54:55 box9 kernel: 2277376 pages of RAM Apr 22 10:54:55 box9 kernel: 284310 reserved pages Apr 22 10:54:55 box9 kernel: 28805905 pages shared Apr 22 10:54:55 box9 kernel: 1 pages swap cached Apr 22 10:54:55 box9 kernel: Out of memory: Killed process 2436, UID 102, (mysqld). Apr 22 10:54:55 box9 kernel: php invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0 Apr 22 10:54:55 box9 kernel: Apr 22 10:54:55 box9 kernel: Call Trace: Apr 22 10:54:55 box9 kernel: [] out_of_memory+0x8e/0x2f3 Apr 22 10:54:55 box9 kernel: [] __wake_up+0x38/0x4f Apr 22 10:54:55 box9 kernel: [] __alloc_pages+0x27f/0x308 Apr 22 10:54:55 box9 kernel: [] :dm_mod:dm_table_any_congested+0x53/0xa3 Apr 22 10:54:56 box9 kernel: [] __do_page_cache_readahead+0x96/0x179 Apr 22 10:54:56 box9 kernel: [] filemap_nopage+0x14c/0x360 Apr 22 10:54:56 box9 kernel: [] __handle_mm_fault+0x1fb/0x1039 Apr 22 10:54:56 box9 kernel: [] do_page_fault+0x4cb/0x874 Apr 22 10:54:56 box9 kernel: [] __up_write+0x27/0xf2 Apr 22 10:54:56 box9 kernel: [] error_exit+0x0/0x84 Apr 22 10:54:56 box9 kernel: Apr 22 10:54:56 box9 kernel: Mem-info: Apr 22 10:54:56 box9 kernel: Node 0 DMA per-cpu: Apr 22 10:54:56 box9 kernel: cpu 0 hot: high 0, batch 1 used:0 Apr 22 10:54:56 box9 kernel: cpu 0 cold: high 0, batch 1 used:0 Apr 22 10:54:56 box9 kernel: cpu 1 hot: high 0, batch 1 used:0 Apr 22 10:54:56 box9 kernel: cpu 1 cold: high 0, batch 1 used:0 Apr 22 10:54:56 box9 kernel: cpu 2 hot: high 0, batch 1 used:0 Apr 22 10:54:56 box9 kernel: cpu 2 cold: high 0, batch 1 used:0 Apr 22 10:54:56 box9 kernel: cpu 3 hot: high 0, batch 1 used:0 Apr 22 10:54:56 box9 kernel: cpu 3 cold: high 0, batch 1 used:0 Apr 22 10:54:56 box9 kernel: Node 0 DMA32 per-cpu: Apr 22 10:54:56 box9 kernel: cpu 0 hot: high 186, batch 31 used:22 Apr 22 10:54:56 box9 kernel: cpu 0 cold: high 62, batch 15 used:14 Apr 22 10:54:56 box9 kernel: cpu 1 hot: high 186, batch 31 used:30 Apr 22 10:54:56 box9 kernel: cpu 1 cold: high 62, batch 15 used:37 Apr 22 10:54:56 box9 kernel: cpu 2 hot: high 186, batch 31 used:24 Apr 22 10:54:56 box9 kernel: cpu 2 cold: high 62, batch 15 used:51 Apr 22 10:54:56 box9 kernel: cpu 3 hot: high 186, batch 31 used:50 Apr 22 10:54:56 box9 kernel: cpu 3 cold: high 62, batch 15 used:54 Apr 22 10:54:56 box9 kernel: Node 0 Normal per-cpu: Apr 22 10:54:56 box9 kernel: cpu 0 hot: high 186, batch 31 used:31 Apr 22 10:54:56 box9 kernel: cpu 0 cold: high 62, batch 15 used:25 Apr 22 10:54:56 box9 kernel: cpu 1 hot: high 186, batch 31 used:26 Apr 22 10:54:56 box9 kernel: cpu 1 cold: high 62, batch 15 used:31 Apr 22 10:54:56 box9 kernel: cpu 2 hot: high 186, batch 31 used:29 Apr 22 10:54:56 box9 kernel: cpu 2 cold: high 62, batch 15 used:48 Apr 22 10:54:56 box9 kernel: cpu 3 hot: high 186, batch 31 used:175 Apr 22 10:54:56 box9 kernel: cpu 3 cold: high 62, batch 15 used:58 Apr 22 10:54:56 box9 kernel: Node 0 HighMem per-cpu: empty Apr 22 10:54:56 box9 kernel: Free pages: 40648kB (0kB HighMem) Apr 22 10:54:56 box9 kernel: Active:1186031 inactive:577442 dirty:0 writeback:0 unstable:0 free:10162 slab:36761 mapped-file:103 mapped-anon:1771555 pagetables:142232 Apr 22 10:54:56 box9 kernel: Node 0 DMA free:10816kB min:12kB low:12kB high:16kB active:0kB inactive:0kB present:10404kB pages_scanned:0 all_unreclaimable? yes Apr 22 10:54:56 box9 kernel: lowmem_reserve[]: 0 3182 7852 7852 Apr 22 10:54:56 box9 kernel: Node 0 DMA32 free:23264kB min:4592kB low:5740kB high:6888kB active:1583408kB inactive:1245212kB present:3258984kB pages_scanned:4761682 all_unreclaimable? yes Apr 22 10:54:56 box9 kernel: lowmem_reserve[]: 0 0 4670 4670 Apr 22 10:54:56 box9 kernel: Node 0 Normal free:6568kB min:6740kB low:8424kB high:10108kB active:3160716kB inactive:1064556kB present:4782464kB pages_scanned:9904014 all_unreclaimable? yes Apr 22 10:54:56 box9 kernel: lowmem_reserve[]: 0 0 0 0 Apr 22 10:54:56 box9 kernel: Node 0 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Apr 22 10:54:56 box9 kernel: lowmem_reserve[]: 0 0 0 0 Apr 22 10:54:56 box9 kernel: Node 0 DMA: 4*4kB 6*8kB 4*16kB 4*32kB 3*64kB 3*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 2*4096kB = 10816kB Apr 22 10:54:56 box9 kernel: Node 0 DMA32: 14*4kB 17*8kB 94*16kB 6*32kB 12*64kB 3*128kB 3*256kB 2*512kB 0*1024kB 1*2048kB 4*4096kB = 23264kB Apr 22 10:54:56 box9 kernel: Node 0 Normal: 10*4kB 134*8kB 41*16kB 14*32kB 10*64kB 1*128kB 2*256kB 2*512kB 2*1024kB 0*2048kB 0*4096kB = 6568kB Apr 22 10:54:56 box9 kernel: Node 0 HighMem: empty Apr 22 10:54:56 box9 kernel: 239 pagecache pages Apr 22 10:54:56 box9 kernel: Swap cache: add 1159956, delete 1159955, find 203407/213236, race 0+46 Apr 22 10:54:56 box9 kernel: Free swap = 0kB Apr 22 10:54:56 box9 kernel: Total swap = 4192924kB Apr 22 10:54:56 box9 kernel: Free swap: 0kB Apr 22 10:54:56 box9 kernel: 2277376 pages of RAM Apr 22 10:54:56 box9 kernel: 284310 reserved pages Apr 22 10:54:56 box9 kernel: 28803760 pages shared Apr 22 10:54:56 box9 kernel: 1 pages swap cached Apr 22 10:54:56 box9 kernel: Out of memory: Killed process 22653, UID 100, (httpd). Apr 22 11:04:33 box9 clamd[2339]: SelfCheck: Database status OK. Apr 22 11:14:33 box9 clamd[2339]: SelfCheck: Database status OK. Apr 22 11:24:33 box9 clamd[2339]: SelfCheck: Database status OK. Apr 22 11:34:33 box9 clamd[2339]: SelfCheck: Database status OK. Apr 22 11:44:33 box9 clamd[2339]: SelfCheck: Database status OK. Apr 22 11:54:33 box9 clamd[2339]: SelfCheck: Database status OK. Apr 22 12:14:31 box9 clamd[2339]: SelfCheck: Database status OK. Apr 22 12:18:20 box9 syslogd 1.4.1: restart. Apr 22 12:18:20 box9 kernel: klogd 1.4.1, log source = /proc/kmsg started. Apr 22 12:18:20 box9 kernel: Linux version 2.6.18-194.32.1.el5 (mockbuild@builder10.centos.org) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-48)) #1 SMP Wed Jan 5 17:52:25 EST 2011