heartbeat[3512]: 2007/05/03_21:46:11 WARN: node wt-vm30: is dead heartbeat[3512]: 2007/05/03_21:46:11 WARN: No STONITH device configured. heartbeat[3512]: 2007/05/03_21:46:11 WARN: Shared disks are not protected. heartbeat[3512]: 2007/05/03_21:46:11 info: Resources being acquired from wt-vm30. heartbeat[3512]: 2007/05/03_21:46:11 info: Link wt-vm30:eth1 dead. harc[3590]: 2007/05/03_21:46:11 info: Running /etc/ha.d/rc.d/status status heartbeat[3591]: 2007/05/03_21:46:12 info: No local resources [/usr/lib/heartbeat/ResourceManager listkeys wt-vm31] to acquire. mach_down[3601]: 2007/05/03_21:46:12 info: Taking over resource group IPaddr::10.3.9.250/24/eth0 ResourceManager[3630]: 2007/05/03_21:46:12 info: Acquiring resource group: wt-vm30 IPaddr::10.3.9.250/24/eth0 drbddisk::drbd0 Filesystem::/dev/drbd0::/srv::ext3 httpd IPaddr[3654]: 2007/05/03_21:46:12 INFO: Resource is stopped ResourceManager[3630]: 2007/05/03_21:46:12 info: Running /etc/ha.d/resource.d/IPaddr 10.3.9.250/24/eth0 start IPaddr[3730]: 2007/05/03_21:46:12 INFO: Using calculated netmask for 10.3.9.250: 255.255.255.0 IPaddr[3730]: 2007/05/03_21:46:12 DEBUG: Using calculated broadcast for 10.3.9.250: 10.3.9.255 IPaddr[3730]: 2007/05/03_21:46:13 INFO: eval /sbin/ifconfig eth0:0 10.3.9.250 netmask 255.255.255.0 broadcast 10.3.9.255 IPaddr[3730]: 2007/05/03_21:46:13 DEBUG: Sending Gratuitous Arp for 10.3.9.250 on eth0:0 [eth0] IPaddr[3709]: 2007/05/03_21:46:13 INFO: Success ResourceManager[3630]: 2007/05/03_21:46:13 info: Running /etc/ha.d/resource.d/drbddisk drbd0 start Filesystem[3861]: 2007/05/03_21:46:13 INFO: Resource is stopped ResourceManager[3630]: 2007/05/03_21:46:13 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /srv ext3 start Filesystem[3902]: 2007/05/03_21:46:13 INFO: Running start for /dev/drbd0 on /srv Filesystem[3899]: 2007/05/03_21:46:13 INFO: Success ResourceManager[3630]: 2007/05/03_21:46:14 info: Running /etc/init.d/httpd start ResourceManager[3630]: 2007/05/03_21:46:14 ERROR: Return code 1 from /etc/init.d/httpd ResourceManager[3630]: 2007/05/03_21:46:14 CRIT: Giving up resources due to failure of httpd ResourceManager[3630]: 2007/05/03_21:46:14 info: Releasing resource group: wt-vm30 IPaddr::10.3.9.250/24/eth0 drbddisk::drbd0 Filesystem::/dev/drbd0::/srv::ext3 httpd ResourceManager[3630]: 2007/05/03_21:46:15 info: Running /etc/init.d/httpd stop ResourceManager[3630]: 2007/05/03_21:46:15 ERROR: Return code 1 from /etc/init.d/httpd ResourceManager[3630]: 2007/05/03_21:46:16 info: Retrying failed stop operation [httpd] ResourceManager[3630]: 2007/05/03_21:46:16 info: Running /etc/init.d/httpd stop ResourceManager[3630]: 2007/05/03_21:46:16 ERROR: Return code 1 from /etc/init.d/httpd ResourceManager[3630]: 2007/05/03_21:46:17 info: Retrying failed stop operation [httpd] ResourceManager[3630]: 2007/05/03_21:46:17 info: Running /etc/init.d/httpd stop ResourceManager[3630]: 2007/05/03_21:46:17 ERROR: Return code 1 from /etc/init.d/httpd ResourceManager[3630]: 2007/05/03_21:46:18 info: Retrying failed stop operation [httpd] ResourceManager[3630]: 2007/05/03_21:46:18 info: Running /etc/init.d/httpd stop ResourceManager[3630]: 2007/05/03_21:46:19 ERROR: Return code 1 from /etc/init.d/httpd ResourceManager[3630]: 2007/05/03_21:46:20 info: Retrying failed stop operation [httpd] ResourceManager[3630]: 2007/05/03_21:46:20 info: Running /etc/init.d/httpd stop ResourceManager[3630]: 2007/05/03_21:46:20 ERROR: Return code 1 from /etc/init.d/httpd ResourceManager[3630]: 2007/05/03_21:46:21 info: Retrying failed stop operation [httpd] ResourceManager[3630]: 2007/05/03_21:46:21 info: Running /etc/init.d/httpd stop ResourceManager[3630]: 2007/05/03_21:46:21 ERROR: Return code 1 from /etc/init.d/httpd ResourceManager[3630]: 2007/05/03_21:46:22 info: Retrying failed stop operation [httpd] ResourceManager[3630]: 2007/05/03_21:46:22 info: Running /etc/init.d/httpd stop ResourceManager[3630]: 2007/05/03_21:46:22 ERROR: Return code 1 from /etc/init.d/httpd ResourceManager[3630]: 2007/05/03_21:46:23 info: Retrying failed stop operation [httpd] ResourceManager[3630]: 2007/05/03_21:46:24 info: Running /etc/init.d/httpd stop ResourceManager[3630]: 2007/05/03_21:46:24 ERROR: Return code 1 from /etc/init.d/httpd ResourceManager[3630]: 2007/05/03_21:46:25 info: Retrying failed stop operation [httpd] ResourceManager[3630]: 2007/05/03_21:46:25 info: Running /etc/init.d/httpd stop ResourceManager[3630]: 2007/05/03_21:46:25 ERROR: Return code 1 from /etc/init.d/httpd ResourceManager[3630]: 2007/05/03_21:46:26 info: Retrying failed stop operation [httpd] ResourceManager[3630]: 2007/05/03_21:46:26 info: Running /etc/init.d/httpd stop ResourceManager[3630]: 2007/05/03_21:46:26 ERROR: Return code 1 from /etc/init.d/httpd ResourceManager[3630]: 2007/05/03_21:46:27 info: Retrying failed stop operation [httpd] ResourceManager[3630]: 2007/05/03_21:46:27 info: Running /etc/init.d/httpd stop ResourceManager[3630]: 2007/05/03_21:46:28 ERROR: Return code 1 from /etc/init.d/httpd ResourceManager[3630]: 2007/05/03_21:46:28 ERROR: Resource script for httpd probably not LSB-compliant. ResourceManager[3630]: 2007/05/03_21:46:28 WARN: it (httpd) MUST succeed on a stop when already stopped ResourceManager[3630]: 2007/05/03_21:46:28 WARN: Machine reboot narrowly avoided! ResourceManager[3630]: 2007/05/03_21:46:28 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /srv ext3 stop Filesystem[4457]: 2007/05/03_21:46:28 INFO: Running stop for /dev/drbd0 on /srv Filesystem[4457]: 2007/05/03_21:46:28 INFO: Trying to unmount /srv Filesystem[4457]: 2007/05/03_21:46:28 INFO: unmounted /srv successfully Filesystem[4454]: 2007/05/03_21:46:28 INFO: Success ResourceManager[3630]: 2007/05/03_21:46:28 info: Running /etc/ha.d/resource.d/drbddisk drbd0 stop ResourceManager[3630]: 2007/05/03_21:46:28 info: Running /etc/ha.d/resource.d/IPaddr 10.3.9.250/24/eth0 stop IPaddr[4565]: 2007/05/03_21:46:29 INFO: /sbin/ifconfig eth0:0 10.3.9.250 down IPaddr[4544]: 2007/05/03_21:46:29 INFO: Success mach_down[3601]: 2007/05/03_21:46:29 info: /usr/lib/heartbeat/mach_down: nice_failback: foreign resources acquired mach_down[3601]: 2007/05/03_21:46:29 info: mach_down takeover complete for node wt-vm30. heartbeat[3512]: 2007/05/03_21:46:29 info: mach_down takeover complete. hb_standby[4613]: 2007/05/03_21:46:59 Going standby [foreign]. heartbeat[3512]: 2007/05/03_21:46:59 info: wt-vm31 wants to go standby [foreign] heartbeat[3512]: 2007/05/03_21:47:10 WARN: No reply to standby request. Standby request cancelled. heartbeat[3512]: 2007/05/03_21:48:46 CRIT: Cluster node wt-vm30 returning after partition. heartbeat[3512]: 2007/05/03_21:48:46 info: For information on cluster partitions, See URL: http://linux-ha.org/SplitBrain heartbeat[3512]: 2007/05/03_21:48:46 WARN: Deadtime value may be too small. heartbeat[3512]: 2007/05/03_21:48:46 info: See FAQ for information on tuning deadtime. heartbeat[3512]: 2007/05/03_21:48:46 info: URL: http://linux-ha.org/FAQ#heavy_load heartbeat[3512]: 2007/05/03_21:48:46 info: Link wt-vm30:eth1 up. heartbeat[3512]: 2007/05/03_21:48:46 WARN: Late heartbeat: Node wt-vm30: interval 184360 ms heartbeat[3512]: 2007/05/03_21:48:46 info: Status update for node wt-vm30: status active harc[4625]: 2007/05/03_21:48:46 info: Running /etc/ha.d/rc.d/status status heartbeat[3512]: 2007/05/03_21:48:46 ERROR: Ignored standby message 'other' from wt-vm30 in state 0 heartbeat[3512]: 2007/05/03_21:48:48 info: Heartbeat shutdown in progress. (3512) heartbeat[4635]: 2007/05/03_21:48:48 info: Giving up all HA resources. ResourceManager[4645]: 2007/05/03_21:48:48 info: Releasing resource group: wt-vm30 IPaddr::10.3.9.250/24/eth0 drbddisk::drbd0 Filesystem::/dev/drbd0::/srv::ext3 httpd ResourceManager[4645]: 2007/05/03_21:48:48 info: Running /etc/init.d/httpd stop ResourceManager[4645]: 2007/05/03_21:48:48 ERROR: Return code 1 from /etc/init.d/httpd ResourceManager[4645]: 2007/05/03_21:48:49 info: Retrying failed stop operation [httpd] ResourceManager[4645]: 2007/05/03_21:48:49 info: Running /etc/init.d/httpd stop ResourceManager[4645]: 2007/05/03_21:48:49 ERROR: Return code 1 from /etc/init.d/httpd ResourceManager[4645]: 2007/05/03_21:48:50 info: Retrying failed stop operation [httpd] ResourceManager[4645]: 2007/05/03_21:48:50 info: Running /etc/init.d/httpd stop ResourceManager[4645]: 2007/05/03_21:48:51 ERROR: Return code 1 from /etc/init.d/httpd ResourceManager[4645]: 2007/05/03_21:48:52 info: Retrying failed stop operation [httpd] ResourceManager[4645]: 2007/05/03_21:48:52 info: Running /etc/init.d/httpd stop ResourceManager[4645]: 2007/05/03_21:48:52 ERROR: Return code 1 from /etc/init.d/httpd ResourceManager[4645]: 2007/05/03_21:48:53 info: Retrying failed stop operation [httpd] ResourceManager[4645]: 2007/05/03_21:48:53 info: Running /etc/init.d/httpd stop ResourceManager[4645]: 2007/05/03_21:48:53 ERROR: Return code 1 from /etc/init.d/httpd ResourceManager[4645]: 2007/05/03_21:48:54 info: Retrying failed stop operation [httpd] ResourceManager[4645]: 2007/05/03_21:48:54 info: Running /etc/init.d/httpd stop ResourceManager[4645]: 2007/05/03_21:48:54 ERROR: Return code 1 from /etc/init.d/httpd ResourceManager[4645]: 2007/05/03_21:48:55 info: Retrying failed stop operation [httpd] ResourceManager[4645]: 2007/05/03_21:48:56 info: Running /etc/init.d/httpd stop ResourceManager[4645]: 2007/05/03_21:48:56 ERROR: Return code 1 from /etc/init.d/httpd ResourceManager[4645]: 2007/05/03_21:48:57 info: Retrying failed stop operation [httpd] ResourceManager[4645]: 2007/05/03_21:48:57 info: Running /etc/init.d/httpd stop ResourceManager[4645]: 2007/05/03_21:48:57 ERROR: Return code 1 from /etc/init.d/httpd ResourceManager[4645]: 2007/05/03_21:48:58 info: Retrying failed stop operation [httpd] ResourceManager[4645]: 2007/05/03_21:48:58 info: Running /etc/init.d/httpd stop ResourceManager[4645]: 2007/05/03_21:48:58 ERROR: Return code 1 from /etc/init.d/httpd ResourceManager[4645]: 2007/05/03_21:48:59 info: Retrying failed stop operation [httpd] ResourceManager[4645]: 2007/05/03_21:48:59 info: Running /etc/init.d/httpd stop ResourceManager[4645]: 2007/05/03_21:49:00 ERROR: Return code 1 from /etc/init.d/httpd ResourceManager[4645]: 2007/05/03_21:49:01 info: Retrying failed stop operation [httpd] ResourceManager[4645]: 2007/05/03_21:49:01 info: Running /etc/init.d/httpd stop ResourceManager[4645]: 2007/05/03_21:49:01 ERROR: Return code 1 from /etc/init.d/httpd ResourceManager[4645]: 2007/05/03_21:49:01 ERROR: Resource script for httpd probably not LSB-compliant. ResourceManager[4645]: 2007/05/03_21:49:01 WARN: it (httpd) MUST succeed on a stop when already stopped ResourceManager[4645]: 2007/05/03_21:49:01 WARN: Machine reboot narrowly avoided! ResourceManager[4645]: 2007/05/03_21:49:01 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /srv ext3 stop Filesystem[5133]: 2007/05/03_21:49:01 INFO: Running stop for /dev/drbd0 on /srv Filesystem[5130]: 2007/05/03_21:49:01 INFO: Success ResourceManager[4645]: 2007/05/03_21:49:01 info: Running /etc/ha.d/resource.d/drbddisk drbd0 stop ResourceManager[4645]: 2007/05/03_21:49:02 info: Running /etc/ha.d/resource.d/IPaddr 10.3.9.250/24/eth0 stop IPaddr[5203]: 2007/05/03_21:49:02 INFO: Success heartbeat[4635]: 2007/05/03_21:49:02 info: All HA resources relinquished. heartbeat[3512]: 2007/05/03_21:49:03 WARN: 1 lost packet(s) for [wt-vm30] [52:54] heartbeat[3512]: 2007/05/03_21:49:03 info: No pkts missing from wt-vm30! heartbeat[3512]: 2007/05/03_21:49:03 info: killing /usr/lib/heartbeat/ipfail process group 3532 with signal 15 heartbeat[3512]: 2007/05/03_21:49:04 info: killing HBFIFO process 3515 with signal 15 heartbeat[3512]: 2007/05/03_21:49:04 info: killing HBWRITE process 3516 with signal 15 heartbeat[3512]: 2007/05/03_21:49:04 info: killing HBREAD process 3517 with signal 15 heartbeat[3512]: 2007/05/03_21:49:04 info: killing HBWRITE process 3518 with signal 15 heartbeat[3512]: 2007/05/03_21:49:04 info: killing HBREAD process 3519 with signal 15 heartbeat[3512]: 2007/05/03_21:49:04 info: Core process 3515 exited. 5 remaining heartbeat[3512]: 2007/05/03_21:49:04 info: Core process 3516 exited. 4 remaining heartbeat[3512]: 2007/05/03_21:49:04 info: Core process 3517 exited. 3 remaining heartbeat[3512]: 2007/05/03_21:49:04 info: Core process 3518 exited. 2 remaining heartbeat[3512]: 2007/05/03_21:49:04 info: Core process 3519 exited. 1 remaining heartbeat[3512]: 2007/05/03_21:49:04 info: wt-vm31 Heartbeat shutdown complete. heartbeat[3512]: 2007/05/03_21:49:04 info: Heartbeat restart triggered. heartbeat[3512]: 2007/05/03_21:49:04 info: Restarting heartbeat. heartbeat[3512]: 2007/05/03_21:49:04 info: Performing heartbeat restart exec. heartbeat[3512]: 2007/05/03_21:49:35 WARN: Logging daemon is disabled --enabling logging daemon is recommended heartbeat[3512]: 2007/05/03_21:49:35 info: ************************** heartbeat[3512]: 2007/05/03_21:49:35 info: Configuration validated. Starting heartbeat 2.0.8 heartbeat[5240]: 2007/05/03_21:49:35 info: heartbeat: version 2.0.8 heartbeat[5240]: 2007/05/03_21:49:35 info: Heartbeat generation: 42 heartbeat[5240]: 2007/05/03_21:49:35 info: G_main_add_TriggerHandler: Added signal manual handler heartbeat[5240]: 2007/05/03_21:49:35 info: G_main_add_TriggerHandler: Added signal manual handler heartbeat[5240]: 2007/05/03_21:49:35 info: Removing /var/run/heartbeat/rsctmp failed, recreating. heartbeat[5240]: 2007/05/03_21:49:35 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1 heartbeat[5240]: 2007/05/03_21:49:35 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1 heartbeat[5240]: 2007/05/03_21:49:35 info: glib: ping heartbeat started. heartbeat[5240]: 2007/05/03_21:49:35 info: G_main_add_SignalHandler: Added signal handler for signal 17 heartbeat[5240]: 2007/05/03_21:49:35 info: Local status now set to: 'up' heartbeat[5240]: 2007/05/03_21:49:36 info: Link wt-vm30:eth1 up. heartbeat[5240]: 2007/05/03_21:49:36 info: Status update for node wt-vm30: status active heartbeat[5240]: 2007/05/03_21:49:36 info: Link 10.3.9.254:10.3.9.254 up. heartbeat[5240]: 2007/05/03_21:49:36 info: Status update for node 10.3.9.254: status ping heartbeat[5240]: 2007/05/03_21:49:36 info: Link wt-vm31:eth1 up. harc[5248]: 2007/05/03_21:49:36 info: Running /etc/ha.d/rc.d/status status heartbeat[5240]: 2007/05/03_21:49:37 info: Comm_now_up(): updating status to active heartbeat[5240]: 2007/05/03_21:49:37 info: Local status now set to: 'active' heartbeat[5240]: 2007/05/03_21:49:37 info: Starting child client "/usr/lib/heartbeat/ipfail" (500,500) heartbeat[5259]: 2007/05/03_21:49:37 info: Starting "/usr/lib/heartbeat/ipfail" as uid 500 gid 500 (pid 5259) heartbeat[5240]: 2007/05/03_21:49:37 info: remote resource transition completed. heartbeat[5240]: 2007/05/03_21:49:37 info: remote resource transition completed. heartbeat[5240]: 2007/05/03_21:49:37 info: Local Resource acquisition completed. (none) heartbeat[5240]: 2007/05/03_21:49:37 info: Initial resource acquisition complete (T_RESOURCES(them))