Dec 8 20:34:12 costello heartbeat[2078]: info: ************************** Dec 8 20:34:12 costello heartbeat[2078]: info: Configuration validated. Starting heartbeat 1.2.3 Dec 8 20:34:12 costello heartbeat[2522]: info: heartbeat: version 1.2.3 Dec 8 20:34:12 costello heartbeat[2522]: info: Heartbeat generation: 257 Dec 8 20:34:12 costello heartbeat[2522]: info: UDP Broadcast heartbeat started on port 694 (694) interface eth0 Dec 8 20:34:12 costello heartbeat[2522]: info: ping heartbeat started. Dec 8 20:34:12 costello heartbeat[2522]: info: pid 2522 locked in memory. Dec 8 20:34:12 costello heartbeat[2522]: info: Local status now set to: 'up' Dec 8 20:34:13 costello heartbeat[2524]: info: pid 2524 locked in memory. Dec 8 20:34:13 costello heartbeat[2525]: info: pid 2525 locked in memory. Dec 8 20:34:13 costello heartbeat[2526]: info: pid 2526 locked in memory. Dec 8 20:34:13 costello heartbeat[2527]: info: pid 2527 locked in memory. Dec 8 20:34:13 costello heartbeat[2528]: info: pid 2528 locked in memory. Dec 8 20:34:13 costello heartbeat[2522]: info: Link abbott:eth0 up. Dec 8 20:34:13 costello heartbeat[2522]: info: Status update for node abbott: status active Dec 8 20:34:13 costello heartbeat[2522]: info: Link costello:eth0 up. Dec 8 20:34:13 costello heartbeat[2522]: ERROR: Exiting HBWRITE process 2527 killed by signal 11. Dec 8 20:34:13 costello heartbeat[2522]: ERROR: Core heartbeat process died! Restarting. Dec 8 20:34:13 costello heartbeat[2522]: WARN: Shutdown delayed until current resource activity finishes. Dec 8 20:34:13 costello heartbeat[2530]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL Dec 8 20:34:13 costello heartbeat: info: Running /etc/ha.d/rc.d/status status Dec 8 20:34:32 costello heartbeat[2522]: WARN: node 192.168.101.1: is dead Dec 8 20:34:32 costello heartbeat[2522]: info: Local status now set to: 'active' Dec 8 20:34:32 costello heartbeat[2522]: info: Starting child client "/usr/lib/heartbeat/ipfail " (1000,102) Dec 8 20:34:32 costello heartbeat[2534]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL Dec 8 20:34:32 costello heartbeat[2535]: info: Starting "/usr/lib/heartbeat/ipfail " as uid 1000 gid 102 (pid 2535) Dec 8 20:34:32 costello heartbeat: info: Running /etc/ha.d/rc.d/status status Dec 8 20:34:32 costello ipfail[2535]: debug: PID=2535 Dec 8 20:34:32 costello ipfail[2535]: debug: Signing in with heartbeat Dec 8 20:34:32 costello ipfail[2535]: debug: [We are costello] Dec 8 20:34:32 costello ipfail[2535]: debug: auto_failback -> 0 (no) Dec 8 20:34:32 costello ipfail[2535]: debug: Setting message filter mode Dec 8 20:34:32 costello ipfail[2535]: debug: Starting node walk Dec 8 20:34:32 costello ipfail[2535]: debug: Cluster node: 192.168.101.1: status: dead Dec 8 20:34:32 costello ipfail[2535]: debug: Cluster node: costello: status: active Dec 8 20:34:32 costello ipfail[2535]: debug: Cluster node: abbott: status: active Dec 8 20:34:32 costello ipfail[2535]: debug: [They are abbott] Dec 8 20:34:32 costello ipfail[2535]: debug: Setting message signal Dec 8 20:34:32 costello ipfail[2535]: debug: Waiting for messages... Dec 8 20:34:33 costello ipfail[2535]: debug: Got asked for num_ping. Dec 8 20:34:33 costello ipfail[2535]: info: Ping node count is balanced. Dec 8 20:34:33 costello ipfail[2535]: debug: Abort message sent. Dec 8 20:34:43 costello heartbeat[2522]: info: Local Resource acquisition completed. (none) Dec 8 20:34:43 costello heartbeat[2522]: info: local resource transition completed. Dec 8 20:34:43 costello heartbeat[2522]: info: remote resource transition completed. Dec 8 20:34:43 costello heartbeat[2522]: info: Heartbeat shutdown in progress. (2522) Dec 8 20:34:43 costello ipfail[2535]: debug: Other side is unstable. Dec 8 20:34:43 costello ipfail[2535]: debug: Other side is now stable. Dec 8 20:34:43 costello heartbeat[2543]: info: Giving up all HA resources. Dec 8 20:34:43 costello ipfail[2535]: debug: Other side is now stable. Dec 8 20:34:43 costello heartbeat: info: Releasing resource group: abbott drbddisk::r0 Filesystem::/dev/drbd0::/home::reiserfs IPaddr2::192.168.101.25/24/eth0 nfs-kernel-server Dec 8 20:34:43 costello heartbeat: info: Running /etc/init.d/nfs-kernel-server stop Dec 8 20:34:43 costello heartbeat: debug: Starting /etc/init.d/nfs-kernel-server stop Dec 8 20:34:43 costello heartbeat: debug: /etc/init.d/nfs-kernel-server stop done. RC=0 Dec 8 20:34:43 costello heartbeat: info: Running /etc/ha.d/resource.d/IPaddr2 192.168.101.25/24/eth0 stop Dec 8 20:34:43 costello heartbeat: debug: Starting /etc/ha.d/resource.d/IPaddr2 192.168.101.25/24/eth0 stop Dec 8 20:34:43 costello heartbeat: debug: /etc/ha.d/resource.d/IPaddr2 192.168.101.25/24/eth0 stop done. RC=0 Dec 8 20:34:43 costello heartbeat: info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /home reiserfs stop Dec 8 20:34:43 costello heartbeat: debug: Starting /etc/ha.d/resource.d/Filesystem /dev/drbd0 /home reiserfs stop Dec 8 20:34:43 costello heartbeat: WARNING: Filesystem /home not mounted? Dec 8 20:34:43 costello heartbeat: debug: /etc/ha.d/resource.d/Filesystem /dev/drbd0 /home reiserfs stop done. RC=0 Dec 8 20:34:43 costello heartbeat: info: Running /etc/ha.d/resource.d/drbddisk r0 stop Dec 8 20:34:43 costello heartbeat: debug: Starting /etc/ha.d/resource.d/drbddisk r0 stop Dec 8 20:34:43 costello heartbeat: debug: /etc/ha.d/resource.d/drbddisk r0 stop done. RC=0 Dec 8 20:34:43 costello heartbeat[2543]: info: killing /usr/lib/heartbeat/ipfail process group 2535 with signal 15 Dec 8 20:34:43 costello heartbeat[2543]: info: All HA resources relinquished. Dec 8 20:34:43 costello heartbeat[2522]: info: killing /usr/lib/heartbeat/ipfail process group 2535 with signal 15 Dec 8 20:34:44 costello heartbeat[2522]: info: killing HBFIFO process 2524 with signal 15 Dec 8 20:34:44 costello heartbeat[2522]: info: killing HBWRITE process 2525 with signal 15 Dec 8 20:34:44 costello heartbeat[2522]: info: killing HBREAD process 2526 with signal 15 Dec 8 20:34:44 costello heartbeat[2522]: info: killing HBREAD process 2528 with signal 15 Dec 8 20:34:44 costello heartbeat[2522]: info: Core process 2528 exited. 4 remaining Dec 8 20:34:44 costello heartbeat[2522]: info: Core process 2526 exited. 3 remaining Dec 8 20:34:44 costello heartbeat[2522]: info: Core process 2525 exited. 2 remaining Dec 8 20:34:44 costello heartbeat[2522]: info: Core process 2524 exited. 1 remaining Dec 8 20:34:44 costello heartbeat[2522]: info: Heartbeat shutdown complete. Dec 8 20:34:44 costello heartbeat[2522]: info: Heartbeat restart triggered. Dec 8 20:34:44 costello heartbeat[2522]: info: Restarting heartbeat. Dec 8 20:34:44 costello heartbeat[2522]: info: Performing heartbeat restart exec. Dec 8 20:34:45 costello kernel: drbd0: Secondary/Secondary --> Secondary/Primary Dec 8 20:34:55 costello heartbeat[2522]: info: ************************** Dec 8 20:34:55 costello heartbeat[2522]: info: Configuration validated. Starting heartbeat 1.2.3 Dec 8 20:34:55 costello heartbeat[2684]: info: heartbeat: version 1.2.3 Dec 8 20:34:55 costello heartbeat[2684]: info: Heartbeat generation: 258 Dec 8 20:34:55 costello heartbeat[2684]: info: UDP Broadcast heartbeat started on port 694 (694) interface eth0 Dec 8 20:34:55 costello heartbeat[2684]: info: ping heartbeat started. Dec 8 20:34:55 costello heartbeat[2684]: info: pid 2684 locked in memory. Dec 8 20:34:55 costello heartbeat[2684]: info: Local status now set to: 'up' Dec 8 20:34:56 costello heartbeat[2686]: info: pid 2686 locked in memory. Dec 8 20:34:56 costello heartbeat[2687]: info: pid 2687 locked in memory. Dec 8 20:34:56 costello heartbeat[2688]: info: pid 2688 locked in memory. Dec 8 20:34:56 costello heartbeat[2684]: info: Link abbott:eth0 up. Dec 8 20:34:56 costello heartbeat[2684]: info: Status update for node abbott: status active Dec 8 20:34:56 costello heartbeat[2684]: info: Link costello:eth0 up. Dec 8 20:34:56 costello heartbeat[2689]: info: pid 2689 locked in memory. Dec 8 20:34:56 costello heartbeat[2684]: ERROR: Exiting HBWRITE process 2689 killed by signal 11. Dec 8 20:34:56 costello heartbeat[2684]: ERROR: Core heartbeat process died! Restarting. Dec 8 20:34:56 costello heartbeat[2684]: WARN: Shutdown delayed until current resource activity finishes. Dec 8 20:34:56 costello heartbeat[2690]: info: pid 2690 locked in memory. Dec 8 20:34:56 costello heartbeat[2691]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL Dec 8 20:34:56 costello heartbeat: info: Running /etc/ha.d/rc.d/status status Dec 8 20:35:16 costello heartbeat[2684]: WARN: node 192.168.101.1: is dead Dec 8 20:35:16 costello heartbeat[2684]: info: Local status now set to: 'active' Dec 8 20:35:16 costello heartbeat[2684]: info: Starting child client "/usr/lib/heartbeat/ipfail " (1000,102) Dec 8 20:35:16 costello heartbeat[2695]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL Dec 8 20:35:16 costello heartbeat: info: Running /etc/ha.d/rc.d/status status Dec 8 20:35:16 costello heartbeat[2696]: info: Starting "/usr/lib/heartbeat/ipfail " as uid 1000 gid 102 (pid 2696) Dec 8 20:35:16 costello ipfail[2696]: debug: PID=2696 Dec 8 20:35:16 costello ipfail[2696]: debug: Signing in with heartbeat Dec 8 20:35:16 costello ipfail[2696]: debug: [We are costello] Dec 8 20:35:16 costello ipfail[2696]: debug: auto_failback -> 0 (no) Dec 8 20:35:16 costello ipfail[2696]: debug: Setting message filter mode Dec 8 20:35:16 costello ipfail[2696]: debug: Starting node walk Dec 8 20:35:16 costello ipfail[2696]: debug: Cluster node: 192.168.101.1: status: dead Dec 8 20:35:16 costello ipfail[2696]: debug: Cluster node: costello: status: active Dec 8 20:35:16 costello ipfail[2696]: debug: Cluster node: abbott: status: active Dec 8 20:35:16 costello ipfail[2696]: debug: [They are abbott] Dec 8 20:35:16 costello ipfail[2696]: debug: Setting message signal Dec 8 20:35:16 costello ipfail[2696]: debug: Waiting for messages... Dec 8 20:35:16 costello ipfail[2696]: debug: Got asked for num_ping. Dec 8 20:35:16 costello ipfail[2696]: info: Ping node count is balanced. Dec 8 20:35:16 costello ipfail[2696]: debug: Abort message sent. Dec 8 20:35:17 costello kernel: drbd0: Secondary/Primary --> Secondary/Secondary Dec 8 20:35:17 costello heartbeat[2684]: info: Received shutdown notice from 'abbott'. Dec 8 20:35:17 costello heartbeat[2684]: info: Resources being acquired from abbott. Dec 8 20:35:17 costello heartbeat[2684]: debug: StartNextRemoteRscReq(): child count 1 Dec 8 20:35:17 costello heartbeat[2704]: info: acquire all HA resources (standby). Dec 8 20:35:17 costello heartbeat[2705]: info: No local resources [/usr/lib/heartbeat/ResourceManager listkeys costello] to acquire. Dec 8 20:35:17 costello heartbeat[2684]: info: Initial resource acquisition complete (T_RESOURCES(us)) Dec 8 20:35:17 costello ipfail[2696]: info: Status update: Node abbott now has status dead Dec 8 20:35:17 costello ipfail[2696]: info: NS: We are dead. :< Dec 8 20:35:17 costello heartbeat[2684]: debug: StartNextRemoteRscReq(): child count 1 Dec 8 20:35:17 costello ipfail[2696]: debug: Other side is unstable. Dec 8 20:35:17 costello heartbeat: info: Acquiring resource group: abbott drbddisk::r0 Filesystem::/dev/drbd0::/home::reiserfs IPaddr2::192.168.101.25/24/eth0 nfs-kernel-server Dec 8 20:35:17 costello heartbeat: info: Running /etc/ha.d/resource.d/drbddisk r0 start Dec 8 20:35:17 costello heartbeat: debug: Starting /etc/ha.d/resource.d/drbddisk r0 start Dec 8 20:35:17 costello kernel: drbd0: Secondary/Secondary --> Primary/Secondary Dec 8 20:35:17 costello heartbeat: debug: /etc/ha.d/resource.d/drbddisk r0 start done. RC=0 Dec 8 20:35:17 costello heartbeat: info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /home reiserfs start Dec 8 20:35:17 costello heartbeat: debug: Starting /etc/ha.d/resource.d/Filesystem /dev/drbd0 /home reiserfs start Dec 8 20:35:17 costello kernel: ReiserFS: drbd0: found reiserfs format "3.6" with standard journal Dec 8 20:35:25 costello kernel: ReiserFS: drbd0: using ordered data mode Dec 8 20:35:25 costello kernel: ReiserFS: drbd0: journal params: device drbd0, size 8192, journal first block 18, max trans len 1024, max batch 900, max commit age 30, max trans age 30 Dec 8 20:35:25 costello kernel: ReiserFS: drbd0: checking transaction log (drbd0) Dec 8 20:35:26 costello kernel: ReiserFS: drbd0: Using r5 hash to sort names Dec 8 20:35:26 costello heartbeat: debug: /etc/ha.d/resource.d/Filesystem /dev/drbd0 /home reiserfs start done. RC=0 Dec 8 20:35:26 costello heartbeat: info: Running /etc/ha.d/resource.d/IPaddr2 192.168.101.25/24/eth0 start Dec 8 20:35:26 costello heartbeat: debug: Starting /etc/ha.d/resource.d/IPaddr2 192.168.101.25/24/eth0 start Dec 8 20:35:26 costello heartbeat: info: /sbin/ip -f inet addr add 192.168.101.25/24 brd 192.168.101.255 dev eth0 Dec 8 20:35:26 costello heartbeat: info: /sbin/ip link set eth0 up Dec 8 20:35:26 costello heartbeat: /usr/lib/heartbeat/send_arp -i 200 -r 5 -p /var/lib/heartbeat/rsctmp/send_arp/send_arp-192.168.101.25 eth0 192.168.101.25 auto 192.168.101.25 ffffffffffff Dec 8 20:35:26 costello heartbeat: debug: /etc/ha.d/resource.d/IPaddr2 192.168.101.25/24/eth0 start done. RC=0 Dec 8 20:35:26 costello heartbeat: info: Running /etc/init.d/nfs-kernel-server start Dec 8 20:35:26 costello heartbeat: debug: Starting /etc/init.d/nfs-kernel-server start Dec 8 20:35:26 costello heartbeat: debug: /etc/init.d/nfs-kernel-server start done. RC=0 Dec 8 20:35:26 costello heartbeat[2704]: info: all HA resource acquisition completed (standby). Dec 8 20:35:26 costello heartbeat[2684]: info: Standby resource acquisition done [all]. Dec 8 20:35:26 costello heartbeat[2945]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL Dec 8 20:35:26 costello heartbeat: info: Running /etc/ha.d/rc.d/status status Dec 8 20:35:26 costello heartbeat: info: Taking over resource group drbddisk::r0 Dec 8 20:35:26 costello heartbeat: info: Acquiring resource group: abbott drbddisk::r0 Filesystem::/dev/drbd0::/home::reiserfs IPaddr2::192.168.101.25/24/eth0 nfs-kernel-server Dec 8 20:35:26 costello heartbeat: info: Running /etc/init.d/nfs-kernel-server start Dec 8 20:35:26 costello heartbeat: debug: Starting /etc/init.d/nfs-kernel-server start Dec 8 20:35:26 costello heartbeat: debug: /etc/init.d/nfs-kernel-server start done. RC=1 Dec 8 20:35:26 costello heartbeat: ERROR: Return code 1 from /etc/init.d/nfs-kernel-server Dec 8 20:35:26 costello heartbeat: info: /usr/lib/heartbeat/mach_down: nice_failback: foreign resources acquired Dec 8 20:35:26 costello heartbeat[2684]: info: Local Resource acquisition completed. (none) Dec 8 20:35:26 costello heartbeat[2684]: info: local resource transition completed. Dec 8 20:35:26 costello heartbeat[2684]: info: mach_down takeover complete. Dec 8 20:35:26 costello heartbeat: info: mach_down takeover complete for node abbott. Dec 8 20:35:26 costello heartbeat[2684]: info: Heartbeat shutdown in progress. (2684) Dec 8 20:35:26 costello heartbeat[3076]: info: Giving up all HA resources. Dec 8 20:35:26 costello heartbeat: info: Releasing resource group: abbott drbddisk::r0 Filesystem::/dev/drbd0::/home::reiserfs IPaddr2::192.168.101.25/24/eth0 nfs-kernel-server Dec 8 20:35:26 costello heartbeat: info: Running /etc/init.d/nfs-kernel-server stop Dec 8 20:35:26 costello heartbeat: debug: Starting /etc/init.d/nfs-kernel-server stop Dec 8 20:35:26 costello rpc.mountd: Caught signal 15, un-registering and exiting. Dec 8 20:35:26 costello heartbeat: debug: /etc/init.d/nfs-kernel-server stop done. RC=0 Dec 8 20:35:26 costello heartbeat: info: Running /etc/ha.d/resource.d/IPaddr2 192.168.101.25/24/eth0 stop Dec 8 20:35:26 costello heartbeat: debug: Starting /etc/ha.d/resource.d/IPaddr2 192.168.101.25/24/eth0 stop Dec 8 20:35:26 costello heartbeat: info: killed previously running send_arp for 192.168.101.25 Dec 8 20:35:26 costello heartbeat: info: /sbin/ip -f inet addr delete 192.168.101.25 dev eth0 Dec 8 20:35:27 costello heartbeat: info: /sbin/ip -o -f inet addr show eth0 Dec 8 20:35:27 costello heartbeat: info: IP Address 192.168.101.25 released Dec 8 20:35:27 costello heartbeat: debug: /etc/ha.d/resource.d/IPaddr2 192.168.101.25/24/eth0 stop done. RC=0 Dec 8 20:35:27 costello heartbeat: info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /home reiserfs stop Dec 8 20:35:27 costello heartbeat: debug: Starting /etc/ha.d/resource.d/Filesystem /dev/drbd0 /home reiserfs stop Dec 8 20:35:27 costello heartbeat: debug: /etc/ha.d/resource.d/Filesystem /dev/drbd0 /home reiserfs stop done. RC=0 Dec 8 20:35:27 costello heartbeat: info: Running /etc/ha.d/resource.d/drbddisk r0 stop Dec 8 20:35:27 costello heartbeat: debug: Starting /etc/ha.d/resource.d/drbddisk r0 stop Dec 8 20:35:27 costello kernel: drbd0: Primary/Secondary --> Secondary/Secondary Dec 8 20:35:27 costello heartbeat: debug: /etc/ha.d/resource.d/drbddisk r0 stop done. RC=0 Dec 8 20:35:27 costello heartbeat[3076]: info: killing /usr/lib/heartbeat/ipfail process group 2696 with signal 15 Dec 8 20:35:27 costello heartbeat[3076]: info: killing heartbeat resource child process group 2945 with signal 9 Dec 8 20:35:27 costello heartbeat[3076]: info: All HA resources relinquished. Dec 8 20:35:27 costello heartbeat[2684]: info: killing /usr/lib/heartbeat/ipfail process group 2696 with signal 15 Dec 8 20:35:27 costello heartbeat[2684]: WARN: node abbott: is dead Dec 8 20:35:27 costello heartbeat[2684]: info: Link abbott:eth0 dead. Dec 8 20:35:28 costello heartbeat[2684]: info: killing HBFIFO process 2686 with signal 15 Dec 8 20:35:28 costello heartbeat[2684]: info: killing HBWRITE process 2687 with signal 15 Dec 8 20:35:28 costello heartbeat[2684]: info: killing HBREAD process 2688 with signal 15 Dec 8 20:35:28 costello heartbeat[2684]: info: killing HBREAD process 2690 with signal 15 Dec 8 20:35:28 costello heartbeat[2684]: info: Core process 2690 exited. 4 remaining Dec 8 20:35:28 costello heartbeat[2684]: info: Core process 2688 exited. 3 remaining Dec 8 20:35:28 costello heartbeat[2684]: info: Core process 2687 exited. 2 remaining Dec 8 20:35:28 costello heartbeat[2684]: info: Core process 2686 exited. 1 remaining Dec 8 20:35:28 costello heartbeat[2684]: info: Heartbeat shutdown complete. Dec 8 20:35:28 costello heartbeat[2684]: info: Heartbeat restart triggered. Dec 8 20:35:28 costello heartbeat[2684]: info: Restarting heartbeat. Dec 8 20:35:28 costello heartbeat[2684]: info: Performing heartbeat restart exec.