[root@wt-vm31 ~]# tail -f /var/log/ha-log
heartbeat[3229]: 2007/05/03_01:17:26 info: Comm_now_up(): updating status to active
heartbeat[3229]: 2007/05/03_01:17:26 info: Local status now set to: 'active'
heartbeat[3229]: 2007/05/03_01:17:26 info: Starting child client "/usr/lib/heartbeat/ipfail" (500,500)
heartbeat[3249]: 2007/05/03_01:17:26 info: Starting "/usr/lib/heartbeat/ipfail" as uid 500  gid 500 (pid 3249)
heartbeat[3229]: 2007/05/03_01:17:27 info: Status update for node wt-vm30: status active
harc[3250]:     2007/05/03_01:17:27 info: Running /etc/ha.d/rc.d/status status
heartbeat[3229]: 2007/05/03_01:17:37 info: local resource transition completed.
heartbeat[3229]: 2007/05/03_01:17:37 info: Initial resource acquisition complete (T_RESOURCES(us))
heartbeat[3260]: 2007/05/03_01:17:37 info: No local resources [/usr/lib/heartbeat/ResourceManager listkeys wt-vm31] to acquire.
heartbeat[3229]: 2007/05/03_01:17:37 info: remote resource transition completed.
heartbeat[3229]: 2007/05/03_01:20:44 WARN: node wt-vm30: is dead
heartbeat[3229]: 2007/05/03_01:20:44 WARN: No STONITH device configured.
heartbeat[3229]: 2007/05/03_01:20:44 WARN: Shared disks are not protected.
heartbeat[3229]: 2007/05/03_01:20:44 info: Resources being acquired from wt-vm30.
heartbeat[3229]: 2007/05/03_01:20:44 info: Link wt-vm30:eth1 dead.
harc[3307]:     2007/05/03_01:20:44 info: Running /etc/ha.d/rc.d/status status
heartbeat[3308]: 2007/05/03_01:20:44 info: No local resources [/usr/lib/heartbeat/ResourceManager listkeys wt-vm31] to acquire.
mach_down[3318]:        2007/05/03_01:20:44 info: Taking over resource group 10.3.9.250
ResourceManager[3347]:  2007/05/03_01:20:44 info: Acquiring resource group: wt-vm30 10.3.9.250 drbddisk::r0 Filesystem::/dev/drbd0::/srv::ext3 httpd
IPaddr[3371]:   2007/05/03_01:20:45 INFO:  Resource is stopped
ResourceManager[3347]:  2007/05/03_01:20:45 info: Running /etc/ha.d/resource.d/IPaddr 10.3.9.250 start
IPaddr[3425]:   2007/05/03_01:20:45 INFO: Using calculated nic for 10.3.9.250: eth0
IPaddr[3425]:   2007/05/03_01:20:45 DEBUG: Using calculated netmask for 10.3.9.250: 255.255.255.0
IPaddr[3425]:   2007/05/03_01:20:45 DEBUG: Using calculated broadcast for 10.3.9.250: 10.3.9.255
IPaddr[3425]:   2007/05/03_01:20:45 INFO: eval /sbin/ifconfig eth0:0 10.3.9.250 netmask 255.255.255.0 broadcast 10.3.9.255
IPaddr[3425]:   2007/05/03_01:20:45 DEBUG: Sending Gratuitous Arp for 10.3.9.250 on eth0:0 [eth0]
IPaddr[3416]:   2007/05/03_01:20:45 INFO:  Success
ResourceManager[3347]:  2007/05/03_01:20:46 info: Running /etc/ha.d/resource.d/drbddisk r0 start
Filesystem[3561]:       2007/05/03_01:20:46 INFO:  Resource is stopped
ResourceManager[3347]:  2007/05/03_01:20:46 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /srv ext3 start
Filesystem[3602]:       2007/05/03_01:20:46 INFO: Running start for /dev/drbd0 on /srv
Filesystem[3599]:       2007/05/03_01:20:46 INFO:  Success
ResourceManager[3347]:  2007/05/03_01:20:46 info: Running /etc/init.d/httpd  start
ResourceManager[3347]:  2007/05/03_01:20:47 ERROR: Return code 1 from /etc/init.d/httpd
ResourceManager[3347]:  2007/05/03_01:20:47 CRIT: Giving up resources due to failure of httpd
ResourceManager[3347]:  2007/05/03_01:20:47 info: Releasing resource group: wt-vm30 10.3.9.250 drbddisk::r0 Filesystem::/dev/drbd0::/srv::ext3 httpd
ResourceManager[3347]:  2007/05/03_01:20:47 info: Running /etc/init.d/httpd  stop
ResourceManager[3347]:  2007/05/03_01:20:47 ERROR: Return code 1 from /etc/init.d/httpd
ResourceManager[3347]:  2007/05/03_01:20:48 info: Retrying failed stop operation [httpd]
ResourceManager[3347]:  2007/05/03_01:20:48 info: Running /etc/init.d/httpd  stop
ResourceManager[3347]:  2007/05/03_01:20:49 ERROR: Return code 1 from /etc/init.d/httpd
ResourceManager[3347]:  2007/05/03_01:20:50 info: Retrying failed stop operation [httpd]
ResourceManager[3347]:  2007/05/03_01:20:50 info: Running /etc/init.d/httpd  stop
ResourceManager[3347]:  2007/05/03_01:20:50 ERROR: Return code 1 from /etc/init.d/httpd
ResourceManager[3347]:  2007/05/03_01:20:51 info: Retrying failed stop operation [httpd]
ResourceManager[3347]:  2007/05/03_01:20:51 info: Running /etc/init.d/httpd  stop
ResourceManager[3347]:  2007/05/03_01:20:51 ERROR: Return code 1 from /etc/init.d/httpd
ResourceManager[3347]:  2007/05/03_01:20:52 info: Retrying failed stop operation [httpd]
ResourceManager[3347]:  2007/05/03_01:20:52 info: Running /etc/init.d/httpd  stop
ResourceManager[3347]:  2007/05/03_01:20:52 ERROR: Return code 1 from /etc/init.d/httpd
ResourceManager[3347]:  2007/05/03_01:20:54 info: Retrying failed stop operation [httpd]
ResourceManager[3347]:  2007/05/03_01:20:54 info: Running /etc/init.d/httpd  stop
ResourceManager[3347]:  2007/05/03_01:20:54 ERROR: Return code 1 from /etc/init.d/httpd
ResourceManager[3347]:  2007/05/03_01:20:55 info: Retrying failed stop operation [httpd]
ResourceManager[3347]:  2007/05/03_01:20:55 info: Running /etc/init.d/httpd  stop
ResourceManager[3347]:  2007/05/03_01:20:55 ERROR: Return code 1 from /etc/init.d/httpd
ResourceManager[3347]:  2007/05/03_01:20:56 info: Retrying failed stop operation [httpd]
ResourceManager[3347]:  2007/05/03_01:20:56 info: Running /etc/init.d/httpd  stop
ResourceManager[3347]:  2007/05/03_01:20:56 ERROR: Return code 1 from /etc/init.d/httpd
ResourceManager[3347]:  2007/05/03_01:20:57 info: Retrying failed stop operation [httpd]
ResourceManager[3347]:  2007/05/03_01:20:57 info: Running /etc/init.d/httpd  stop
ResourceManager[3347]:  2007/05/03_01:20:58 ERROR: Return code 1 from /etc/init.d/httpd
ResourceManager[3347]:  2007/05/03_01:20:59 info: Retrying failed stop operation [httpd]
ResourceManager[3347]:  2007/05/03_01:20:59 info: Running /etc/init.d/httpd  stop
ResourceManager[3347]:  2007/05/03_01:20:59 ERROR: Return code 1 from /etc/init.d/httpd
ResourceManager[3347]:  2007/05/03_01:21:00 info: Retrying failed stop operation [httpd]
ResourceManager[3347]:  2007/05/03_01:21:00 info: Running /etc/init.d/httpd  stop
ResourceManager[3347]:  2007/05/03_01:21:00 ERROR: Return code 1 from /etc/init.d/httpd
ResourceManager[3347]:  2007/05/03_01:21:00 ERROR: Resource script for httpd probably not LSB-compliant.
ResourceManager[3347]:  2007/05/03_01:21:00 WARN: it (httpd) MUST succeed on a stop when already stopped
ResourceManager[3347]:  2007/05/03_01:21:00 WARN: Machine reboot narrowly avoided!
ResourceManager[3347]:  2007/05/03_01:21:01 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /srv ext3 stop
Filesystem[4157]:       2007/05/03_01:21:01 INFO: Running stop for /dev/drbd0 on /srv
Filesystem[4157]:       2007/05/03_01:21:01 INFO: Trying to unmount /srv
Filesystem[4157]:       2007/05/03_01:21:01 INFO: unmounted /srv successfully
Filesystem[4154]:       2007/05/03_01:21:01 INFO:  Success
ResourceManager[3347]:  2007/05/03_01:21:01 info: Running /etc/ha.d/resource.d/drbddisk r0 stop
ResourceManager[3347]:  2007/05/03_01:21:01 info: Running /etc/ha.d/resource.d/IPaddr 10.3.9.250 stop
IPaddr[4255]:   2007/05/03_01:21:01 INFO: /sbin/ifconfig eth0:0 10.3.9.250 down
IPaddr[4246]:   2007/05/03_01:21:01 INFO:  Success
mach_down[3318]:        2007/05/03_01:21:01 info: /usr/lib/heartbeat/mach_down: nice_failback: foreign resources acquired
mach_down[3318]:        2007/05/03_01:21:01 info: mach_down takeover complete for node wt-vm30.
heartbeat[3229]: 2007/05/03_01:21:01 info: mach_down takeover complete.
hb_standby[4304]:       2007/05/03_01:21:31 Going standby [foreign].
heartbeat[3229]: 2007/05/03_01:21:32 info: wt-vm31 wants to go standby [foreign]
heartbeat[3229]: 2007/05/03_01:21:43 WARN: No reply to standby request.  Standby request cancelled.