drbd.conf: ========= global { usage-count yes; } common { syncer { rate 10M; } } resource r0 { protocol C; net { cram-hmac-alg sha1; shared-secret "FooFunFactory"; } on wt-vm30 { device /dev/drbd0; disk /dev/sda5; address 192.168.0.3:7788; meta-disk internal; } on wt-vm31 { device /dev/drbd0; disk /dev/sda5; address 192.168.0.4:7788; meta-disk internal; } } ha.cf: ====== debugfile /var/log/ha-debug logfile /var/log/ha-log logfacility local0 keepalive 2 deadtime 30 warntime 10 initdead 120 udpport 694 bcast eth1 auto_failback on node wt-vm30 node wt-vm31 ping 10.3.9.254 #respawn userid /path/name/to/run respawn hacluster /usr/lib/heartbeat/ipfail #apiauth client-name gid=gidlist uid=uidlist #apiauth ipfail gid=haclient uid=hacluster #crm on haresources: ============ wt-vm30 \ drbddisk::r0 \ Filesystem::/dev/drbd0::/drbd::ext3 \ apache2::/etc/apache2/httpd.conf \ IPaddr2::10.3.9.250/24/eth0 ha-log from wt-vm31: ==================== tail -f /var/log/ha-log heartbeat[18236]: 2007/05/30_16:23:26 WARN: Core dumps could be lost if multiple dumps occur heartbeat[18236]: 2007/05/30_16:23:26 WARN: Consider setting /proc/sys/kernel/core_uses_pid (or equivalent) to 1 for maximum supportability heartbeat[18236]: 2007/05/30_16:23:26 info: Version 2 support: false heartbeat[18236]: 2007/05/30_16:23:26 WARN: Logging daemon is disabled --enabling logging daemon is recommended heartbeat[18236]: 2007/05/30_16:23:26 info: ************************** heartbeat[18236]: 2007/05/30_16:23:26 info: Configuration validated. Starting heartbeat 2.0.9 heartbeat[18237]: 2007/05/30_16:23:26 info: heartbeat: version 2.0.9 heartbeat[18237]: 2007/05/30_16:23:26 info: Heartbeat generation: 17 heartbeat[18237]: 2007/05/30_16:23:26 info: G_main_add_TriggerHandler: Added signal manual handler heartbeat[18237]: 2007/05/30_16:23:26 info: G_main_add_TriggerHandler: Added signal manual handler heartbeat[18237]: 2007/05/30_16:23:26 info: Removing /var/run/heartbeat/rsctmp failed, recreating. heartbeat[18237]: 2007/05/30_16:23:26 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1 heartbeat[18237]: 2007/05/30_16:23:26 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1 heartbeat[18237]: 2007/05/30_16:23:26 info: glib: ping heartbeat started. heartbeat[18237]: 2007/05/30_16:23:26 info: G_main_add_SignalHandler: Added signal handler for signal 17 heartbeat[18237]: 2007/05/30_16:23:26 info: Local status now set to: 'up' heartbeat[18237]: 2007/05/30_16:23:27 info: Link 10.3.9.254:10.3.9.254 up. heartbeat[18237]: 2007/05/30_16:23:27 info: Status update for node 10.3.9.254: status ping heartbeat[18237]: 2007/05/30_16:23:27 info: Link wt-vm30:eth1 heartbeat[18237]: 2007/05/30_16:23:54 info: Link wt-vm31:eth1 up. heartbeat[18237]: 2007/05/30_16:23:54 info: Status update for node wt-vm31: status up harc[18245]: 2007/05/30_16:23:55 info: Running /etc/ha.d/rc.d/status status heartbeat[18237]: 2007/05/30_16:23:55 ERROR: HBDoMsg_T_REPNODES: Attempt to delete node 10.3.9.254 heartbeat[18237]: 2007/05/30_16:23:55 info: Comm_now_up(): updating status to active heartbeat[18237]: 2007/05/30_16:23:55 info: Local status now set to: 'active' heartbeat[18237]: 2007/05/30_16:23:55 info: Starting child client "/usr/lib/heartbeat/ipfail" (90,90) heartbeat[18256]: 2007/05/30_16:23:55 info: Starting "/usr/lib/heartbeat/ipfail" as uid 90 gid 90 (pid 18256) heartbeat[18237]: 2007/05/30_16:23:55 info: Status update for node wt-vm31: status active harc[18257]: 2007/05/30_16:23:56 info: Running /etc/ha.d/rc.d/status status heartbeat[18237]: 2007/05/30_16:24:06 info: local resource transition completed. heartbeat[18237]: 2007/05/30_16:24:06 info: Initial resource acquisition complete (T_RESOURCES(us)) heartbeat[18267]: 2007/05/30_16:24:06 info: Local Resource acquisition completed. harc[18300]: 2007/05/30_16:24:06 info: Running /etc/ha.d/rc.d/ip-request-resp ip-request-resp ip-request-resp[18300]: 2007/05/30_16:24:06 received ip-request-resp drbddisk::r0 OK yes ResourceManager[18315]: 2007/05/30_16:24:06 info: Acquiring resource group: wt-vm30 drbddisk::r0 Filesystem::/dev/drbd0::/drbd::ext3 apache2 # apache2::/etc/apache2/httpd.conf IPaddr2::10.3.9.250/24/eth0 ResourceManager[18315]: 2007/05/30_16:24:06 info: Running /etc/ha.d/resource.d/drbddisk r0 start heartbeat[18237]: 2007/05/30_16:24:06 info: remote resource transition completed. Filesystem[18380]: 2007/05/30_16:24:06 INFO: Resource is stopped ResourceManager[18315]: 2007/05/30_16:24:06 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /drbd ext3 start Filesystem[18425]: 2007/05/30_16:24:06 INFO: Running start for /dev/drbd0 on /drbd Filesystem[18420]: 2007/05/30_16:24:06 INFO: Success ResourceManager[18315]: 2007/05/30_16:24:06 info: Running /etc/init.d/apache2 start ResourceManager[18315]: 2007/05/30_16:24:09 ERROR: Cannot locate resource script # ResourceManager[18315]: 2007/05/30_16:24:09 ERROR: Cannot locate resource script # ResourceManager[18315]: 2007/05/30_16:24:09 ERROR: Cannot locate resource script # ResourceManager[18315]: 2007/05/30_16:24:09 CRIT: Giving up resources due to failure of # ResourceManager[18315]: 2007/05/30_16:24:09 info: Releasing resource group: wt-vm30 drbddisk::r0 Filesystem::/dev/drbd0::/drbd::ext3 apache2 # apache2::/etc/apache2/httpd.conf IPaddr2::10.3.9.250/24/eth0 ResourceManager[18315]: 2007/05/30_16:24:09 info: Running /etc/ha.d/resource.d/IPaddr2 10.3.9.250/24/eth0 stop IPaddr2[18596]: 2007/05/30_16:24:09 ERROR: [/usr/lib/heartbeat/findif -C] failed IPaddr2[18573]: 2007/05/30_16:24:09 ERROR: Illegal argument ResourceManager[18315]: 2007/05/30_16:24:09 ERROR: Return code 2 from /etc/ha.d/resource.d/IPaddr2 ResourceManager[18315]: 2007/05/30_16:24:10 info: Retrying failed stop operation [IPaddr2::10.3.9.250/24/eth0] ResourceManager[18315]: 2007/05/30_16:24:10 info: Running /etc/ha.d/resource.d/IPaddr2 10.3.9.250/24/eth0 stop IPaddr2[18671]: 2007/05/30_16:24:10 ERROR: [/usr/lib/heartbeat/findif -C] failed IPaddr2[18648]: 2007/05/30_16:24:10 ERROR: Illegal argument ResourceManager[18315]: 2007/05/30_16:24:10 ERROR: Return code 2 from /etc/ha.d/resource.d/IPaddr2 ResourceManager[18315]: 2007/05/30_16:24:11 info: Retrying failed stop operation [IPaddr2::10.3.9.250/24/eth0] ResourceManager[18315]: 2007/05/30_16:24:11 info: Running /etc/ha.d/resource.d/IPaddr2 10.3.9.250/24/eth0 stop IPaddr2[18746]: 2007/05/30_16:24:12 ERROR: [/usr/lib/heartbeat/findif -C] failed IPaddr2[18723]: 2007/05/30_16:24:12 ERROR: Illegal argument ResourceManager[18315]: 2007/05/30_16:24:12 ERROR: Return code 2 from /etc/ha.d/resource.d/IPaddr2 ResourceManager[18315]: 2007/05/30_16:24:13 info: Retrying failed stop operation [IPaddr2::10.3.9.250/24/eth0] ResourceManager[18315]: 2007/05/30_16:24:13 info: Running /etc/ha.d/resource.d/IPaddr2 10.3.9.250/24/eth0 stop IPaddr2[18821]: 2007/05/30_16:24:13 ERROR: [/usr/lib/heartbeat/findif -C] failed IPaddr2[18798]: 2007/05/30_16:24:13 ERROR: Illegal argument ResourceManager[18315]: 2007/05/30_16:24:13 ERROR: Return code 2 from /etc/ha.d/resource.d/IPaddr2 ResourceManager[18315]: 2007/05/30_16:24:14 info: Retrying failed stop operation [IPaddr2::10.3.9.250/24/eth0] ResourceManager[18315]: 2007/05/30_16:24:14 info: Running /etc/ha.d/resource.d/IPaddr2 10.3.9.250/24/eth0 stop IPaddr2[18896]: 2007/05/30_16:24:14 ERROR: [/usr/lib/heartbeat/findif -C] failed IPaddr2[18873]: 2007/05/30_16:24:14 ERROR: Illegal argument ResourceManager[18315]: 2007/05/30_16:24:14 ERROR: Return code 2 from /etc/ha.d/resource.d/IPaddr2 up. wt-vm30:~ # tail -f /var/log/ha-log heartbeat[22455]: 2007/05/30_16:27:51 info: G_main_add_TriggerHandler: Added signal manual handler heartbeat[22455]: 2007/05/30_16:27:51 info: Removing /var/run/heartbeat/rsctmp failed, recreating. heartbeat[22455]: 2007/05/30_16:27:51 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1 heartbeat[22455]: 2007/05/30_16:27:51 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1 heartbeat[22455]: 2007/05/30_16:27:51 info: glib: ping heartbeat started. heartbeat[22455]: 2007/05/30_16:27:51 info: G_main_add_SignalHandler: Added signal handler for signal 17 heartbeat[22455]: 2007/05/30_16:27:51 info: Local status now set to: 'up' heartbeat[22455]: 2007/05/30_16:27:52 info: Link 10.3.9.254:10.3.9.254 up. heartbeat[22455]: 2007/05/30_16:27:52 info: Status update for node 10.3.9.254: status ping heartbeat[22455]: 2007/05/30_16:27:52 info: Link wt-vm30:eth1 up. heartbeat[22455]: 2007/05/30_16:28:07 info: Link wt-vm31:eth1 up. heartbeat[22455]: 2007/05/30_16:28:07 info: Status update for node wt-vm31: status up harc[22464]: 2007/05/30_16:28:07 info: Running /etc/ha.d/rc.d/status status heartbeat[22455]: 2007/05/30_16:28:07 ERROR: HBDoMsg_T_REPNODES: Attempt to delete node 10.3.9.254 heartbeat[22455]: 2007/05/30_16:28:07 info: Comm_now_up(): updating status to active heartbeat[22455]: 2007/05/30_16:28:07 info: Local status now set to: 'active' heartbeat[22455]: 2007/05/30_16:28:07 info: Starting child client "/usr/lib/heartbeat/ipfail" (90,90) heartbeat[22475]: 2007/05/30_16:28:07 info: Starting "/usr/lib/heartbeat/ipfail" as uid 90 gid 90 (pid 22475) heartbeat[22455]: 2007/05/30_16:28:08 info: Status update for node wt-vm31: status active harc[22476]: 2007/05/30_16:28:08 info: Running /etc/ha.d/rc.d/status status ip-request-resp[22519]: 2007/05/30_16:28:19 received ip-request-resp drbddisk::r0 OK yes ResourceManager[22534]: 2007/05/30_16:28:19 info: Acquiring resource group: wt-vm30 drbddisk::r0 Filesystem::/dev/drbd0::/drbd::ext3 apache2::/etc/apache2/httpd.conf IPaddr::10.3.9.250/24/eth0 ResourceManager[22534]: 2007/05/30_16:28:19 info: Running /etc/ha.d/resource.d/drbddisk r0 start Filesystem[22599]: 2007/05/30_16:28:19 INFO: Resource is stopped ResourceManager[22534]: 2007/05/30_16:28:19 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /drbd ext3 start Filesystem[22644]: 2007/05/30_16:28:19 INFO: Running start for /dev/drbd0 on /drbd Filesystem[22639]: 2007/05/30_16:28:19 INFO: Success ResourceManager[22534]: 2007/05/30_16:28:19 info: Running /etc/init.d/apache2 /etc/apache2/httpd.conf start ResourceManager[22534]: 2007/05/30_16:28:19 ERROR: Return code 1 from /etc/init.d/apache2 ResourceManager[22534]: 2007/05/30_16:28:19 CRIT: Giving up resources due to failure of apache2::/etc/apache2/httpd.conf ResourceManager[22534]: 2007/05/30_16:28:19 info: Releasing resource group: wt-vm30 drbddisk::r0 Filesystem::/dev/drbd0::/drbd::ext3 apache2::/etc/apache2/httpd.conf IPaddr::10.3.9.250/24/eth0 ResourceManager[22534]: 2007/05/30_16:28:19 info: Running /etc/ha.d/resource.d/IPaddr 10.3.9.250/24/eth0 stop IPaddr[22777]: 2007/05/30_16:28:20 INFO: Success ResourceManager[22534]: 2007/05/30_16:28:20 info: Running /etc/init.d/apache2 /etc/apache2/httpd.conf stop ResourceManager[22534]: 2007/05/30_16:28:20 ERROR: Return code 1 from /etc/init.d/apache2 ResourceManager[22534]: 2007/05/30_16:28:21 info: Retrying failed stop operation [apache2::/etc/apache2/httpd.conf] ResourceManager[22534]: 2007/05/30_16:28:21 info: Running /etc/init.d/apache2 /etc/apache2/httpd.conf stop ResourceManager[22534]: 2007/05/30_16:28:21 ERROR: Return code 1 from /etc/init.d/apache2 ResourceManager[22534]: 2007/05/30_16:28:22 info: Retrying failed stop operation [apache2::/etc/apache2/httpd.conf] ResourceManager[22534]: 2007/05/30_16:28:22 info: Running /etc/init.d/apache2 /etc/apache2/httpd.conf stop ResourceManager[22534]: 2007/05/30_16:28:22 ERROR: Return code 1 from /etc/init.d/apache2 ResourceManager[22534]: 2007/05/30_16:28:23 info: Retrying failed stop operation [apache2::/etc/apache2/httpd.conf] ResourceManager[22534]: 2007/05/30_16:28:23 info: Running /etc/init.d/apache2 /etc/apache2/httpd.conf stop ResourceManager[22534]: 2007/05/30_16:28:23 ERROR: Return code 1 from /etc/init.d/apache2 ResourceManager[22534]: 2007/05/30_16:28:24 info: Retrying failed stop operation [apache2::/etc/apache2/httpd.conf] ResourceManager[22534]: 2007/05/30_16:28:25 info: Running /etc/init.d/apache2 /etc/apache2/httpd.conf stop ResourceManager[22534]: 2007/05/30_16:28:25 ERROR: Return code 1 from /etc/init.d/apache2 ResourceManager[22534]: 2007/05/30_16:28:26 info: Retrying failed stop operation [apache2::/etc/apache2/httpd.conf] ResourceManager[22534]: 2007/05/30_16:28:26 info: Running /etc/init.d/apache2 /etc/apache2/httpd.conf stop ResourceManager[22534]: 2007/05/30_16:28:26 ERROR: Return code 1 from /etc/init.d/apache2 ResourceManager[22534]: 2007/05/30_16:28:27 info: Retrying failed stop operation [apache2::/etc/apache2/httpd.conf] ResourceManager[22534]: 2007/05/30_16:28:27 info: Running /etc/init.d/apache2 /etc/apache2/httpd.conf stop ResourceManager[22534]: 2007/05/30_16:28:27 ERROR: Return code 1 from /etc/init.d/apache2 ResourceManager[22534]: 2007/05/30_16:28:28 info: Retrying failed stop operation [apache2::/etc/apache2/httpd.conf] ResourceManager[22534]: 2007/05/30_16:28:28 info: Running /etc/init.d/apache2 /etc/apache2/httpd.conf stop ResourceManager[22534]: 2007/05/30_16:28:28 ERROR: Return code 1 from /etc/init.d/apache2 ResourceManager[22534]: 2007/05/30_16:28:32 ERROR: Return code 1 from /etc/init.d/apache2 ResourceManager[22534]: 2007/05/30_16:28:32 ERROR: Resource script for apache2::/etc/apache2/httpd.conf probably not LSB-compliant. ResourceManager[22534]: 2007/05/30_16:28:32 WARN: it (apache2::/etc/apache2/httpd.conf) MUST succeed on a stop when already stopped ResourceManager[22534]: 2007/05/30_16:28:32 WARN: Machine reboot narrowly avoided! ResourceManager[22534]: 2007/05/30_16:28:32 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /drbd ext3 stop Filesystem[23449]: 2007/05/30_16:28:32 INFO: Running stop for /dev/drbd0 on /drbd Filesystem[23449]: 2007/05/30_16:28:32 INFO: Trying to unmount /drbd Filesystem[23449]: 2007/05/30_16:28:32 INFO: unmounted /drbd successfully Filesystem[23444]: 2007/05/30_16:28:32 INFO: Success ResourceManager[22534]: 2007/05/30_16:28:32 info: Running /etc/ha.d/resource.d/drbddisk r0 stop hb_standby[23527]: 2007/05/30_16:29:02 Going standby [foreign]. heartbeat[22455]: 2007/05/30_16:29:02 info: wt-vm30 wants to go standby [foreign] heartbeat[22455]: 2007/05/30_16:29:03 info: standby: wt-vm31 can take our foreign resources heartbeat[23538]: 2007/05/30_16:29:03 info: give up foreign HA resources (standby). heartbeat[23538]: 2007/05/30_16:29:03 info: foreign HA resource release completed (standby). heartbeat[22455]: 2007/05/30_16:29:03 info: Local standby process completed [foreign]. heartbeat[22455]: 2007/05/30_16:29:04 WARN: 1 lost packet(s) for [wt-vm31] [43:45] heartbeat[22455]: 2007/05/30_16:29:04 info: remote resource transition completed. heartbeat[22455]: 2007/05/30_16:29:04 info: No pkts missing from wt-vm31! heartbeat[22455]: 2007/05/30_16:29:04 info: Other node completed standby takeover of foreign resources.