-----------crm_mon---------------------------- Refresh in 2s... ============ Last updated: Fri Apr 4 13:23:51 2008 Current DC: apolo-slave (30fcbb73-aabd-4d15-8f71-bfb9e538885c) 2 Nodes configured. 3 Resources configured. ============ Node: apolo-slave (30fcbb73-aabd-4d15-8f71-bfb9e538885c): online Node: apolo-master (23e734e2-b052-429a-b5ae-c3d31c4487a1): online Master/Slave Set: ms-drbd0 drbd0:0 (heartbeat::ocf:drbd): Started apolo-slave drbd0:1 (heartbeat::ocf:drbd): Master apolo-master FAILED Resource Group: group_1 IPaddr_192_168_2_150 (heartbeat::ocf:IPaddr): Started apolo-master ssh_2 (lsb:ssh): Started apolo-master Clone Set: stonithclone stonithclone:0 (stonith:external/ssh): Started apolo-slave stonithclone:1 (stonith:external/ssh): Started apolo-master Failed actions: drbd0:1_promote_0 (node=apolo-master, call=97, rc=1): complete ----------------------------hb log-------------------------- crmd[26524]: 2008/04/04_14:25:27 info: process_lrm_event: LRM operation drbd0:1_demote_0 (call=149, rc=0) complete crmd[26524]: 2008/04/04_14:25:28 info: do_lrm_rsc_op: Performing op=drbd0:1_notify_0 key=73:34:2987022a-c9a2-4803-a95c-d3a20c973d23) lrmd[26521]: 2008/04/04_14:25:28 info: rsc:drbd0:1: notify crm_master[962]: 2008/04/04_14:25:28 info: Invoked: /usr/sbin/crm_master -v 75 -l reboot lrmd[26521]: 2008/04/04_14:25:29 info: RA output: (drbd0:1:notify:stdout) No set matching id=master-23e734e2-b052-429a-b5ae-c3d31c4487a1 in status crmd[26524]: 2008/04/04_14:25:29 info: process_lrm_event: LRM operation drbd0:1_notify_0 (call=150, rc=0) complete crmd[26524]: 2008/04/04_14:25:30 info: do_lrm_rsc_op: Performing op=drbd0:1_notify_0 key=67:35:2987022a-c9a2-4803-a95c-d3a20c973d23) lrmd[26521]: 2008/04/04_14:25:30 info: rsc:drbd0:1: notify crmd[26524]: 2008/04/04_14:25:30 info: process_lrm_event: LRM operation drbd0:1_notify_0 (call=151, rc=0) complete crmd[26524]: 2008/04/04_14:25:32 info: do_lrm_rsc_op: Performing op=drbd0:1_stop_0 key=3:35:2987022a-c9a2-4803-a95c-d3a20c973d23) lrmd[26521]: 2008/04/04_14:25:32 info: rsc:drbd0:1: stop crm_master[1045]: 2008/04/04_14:25:32 info: Invoked: /usr/sbin/crm_master -D lrmd[26521]: 2008/04/04_14:25:33 info: RA output: (drbd0:1:stop:stdout) lrmd[26521]: 2008/04/04_14:25:33 info: RA output: (drbd0:1:stop:stdout) crmd[26524]: 2008/04/04_14:25:33 info: process_lrm_event: LRM operation drbd0:1_stop_0 (call=152, rc=0) complete crmd[26524]: 2008/04/04_14:25:34 info: do_lrm_rsc_op: Performing op=drbd0:0_monitor_0 key=8:36:2987022a-c9a2-4803-a95c-d3a20c973d23) lrmd[26521]: 2008/04/04_14:25:34 info: rsc:drbd0:0: monitor crmd[26524]: 2008/04/04_14:25:34 info: do_lrm_rsc_op: Performing op=IPaddr_192_168_2_150_start_0 key=40:36:2987022a-c9a2-4803-a95c-d3a20c973d23) lrmd[26521]: 2008/04/04_14:25:34 info: rsc:IPaddr_192_168_2_150: start IPaddr[1109]: 2008/04/04_14:25:34 INFO: Using calculated nic for 192.168.2.150: eth0 IPaddr[1109]: 2008/04/04_14:25:34 INFO: Using calculated netmask for 192.168.2.150: 255.255.255.0 crmd[26524]: 2008/04/04_14:25:34 info: process_lrm_event: LRM operation drbd0:0_monitor_0 (call=153, rc=7) complete IPaddr[1109]: 2008/04/04_14:25:34 INFO: eval ifconfig eth0:0 192.168.2.150 netmask 255.255.255.0 broadcast 192.168.2.255 crmd[26524]: 2008/04/04_14:25:34 info: process_lrm_event: LRM operation IPaddr_192_168_2_150_start_0 (call=154, rc=0) complete crmd[26524]: 2008/04/04_14:25:36 info: do_lrm_rsc_op: Performing op=drbd0:1_start_0 key=10:36:2987022a-c9a2-4803-a95c-d3a20c973d23) lrmd[26521]: 2008/04/04_14:25:36 info: rsc:drbd0:1: start crmd[26524]: 2008/04/04_14:25:36 info: do_lrm_rsc_op: Performing op=IPaddr_192_168_2_150_monitor_5000 key=41:36:2987022a-c9a2-4803-a95c-d3a20c973d23) crmd[26524]: 2008/04/04_14:25:36 info: do_lrm_rsc_op: Performing op=ssh_2_start_0 key=43:36:2987022a-c9a2-4803-a95c-d3a20c973d23) lrmd[26521]: 2008/04/04_14:25:36 info: rsc:ssh_2: start lrmd[1251]: 2008/04/04_14:25:36 WARN: For LSB init script, no additional parameters are needed. lrmd[26521]: 2008/04/04_14:25:36 info: RA output: (ssh_2:start:stdout) Starting OpenBSD Secure Shell server... lrmd[26521]: 2008/04/04_14:25:36 info: RA output: (ssh_2:start:stdout) . crmd[26524]: 2008/04/04_14:25:36 info: process_lrm_event: LRM operation ssh_2_start_0 (call=157, rc=0) complete crmd[26524]: 2008/04/04_14:25:36 info: process_lrm_event: LRM operation IPaddr_192_168_2_150_monitor_5000 (call=156, rc=0) complete crmd[26524]: 2008/04/04_14:25:37 info: do_lrm_rsc_op: Performing op=ssh_2_monitor_120000 key=44:36:2987022a-c9a2-4803-a95c-d3a20c973d23) crmd[26524]: 2008/04/04_14:25:37 info: process_lrm_event: LRM operation ssh_2_monitor_120000 (call=158, rc=0) complete lrmd[26521]: 2008/04/04_14:25:37 info: RA output: (drbd0:1:start:stdout) crmd[26524]: 2008/04/04_14:25:37 info: process_lrm_event: LRM operation drbd0:1_start_0 (call=155, rc=0) complete crmd[26524]: 2008/04/04_14:25:39 info: do_lrm_rsc_op: Performing op=drbd0:1_notify_0 key=61:36:2987022a-c9a2-4803-a95c-d3a20c973d23) lrmd[26521]: 2008/04/04_14:25:39 info: rsc:drbd0:1: notify crm_master[1459]: 2008/04/04_14:25:39 info: Invoked: /usr/sbin/crm_master -v 10 -l reboot lrmd[26521]: 2008/04/04_14:25:40 info: RA output: (drbd0:1:notify:stdout) No set matching id=master-23e734e2-b052-429a-b5ae-c3d31c4487a1 in status crmd[26524]: 2008/04/04_14:25:40 info: process_lrm_event: LRM operation drbd0:1_notify_0 (call=159, rc=0) complete crmd[26524]: 2008/04/04_14:25:41 info: do_lrm_rsc_op: Performing op=drbd0:1_notify_0 key=62:37:2987022a-c9a2-4803-a95c-d3a20c973d23) lrmd[26521]: 2008/04/04_14:25:41 info: rsc:drbd0:1: notify crmd[26524]: 2008/04/04_14:25:41 info: process_lrm_event: LRM operation drbd0:1_notify_0 (call=160, rc=0) complete crmd[26524]: 2008/04/04_14:25:42 info: do_lrm_rsc_op: Performing op=drbd0:1_promote_0 key=12:37:2987022a-c9a2-4803-a95c-d3a20c973d23) lrmd[26521]: 2008/04/04_14:25:42 info: rsc:drbd0:1: promote lrmd[26521]: 2008/04/04_14:25:42 info: RA output: (drbd0:1:promote:stdout) ioctl(,SET_STATE,) failed: Input/output error Local replica is inconsistent (--do-what-I-say ?) Command 'drbdsetup /dev/drbd0 primary' terminated with exit code 21 /bin/bash: -c: option requires an argument Command '/bin/bash -c' terminated with exit code 2 drbdadm aborting drbd[1492]: 2008/04/04_14:25:42 ERROR: drbd0 promote: Not primary despite drbdadm call. crmd[26524]: 2008/04/04_14:25:42 info: process_lrm_event: LRM operation drbd0:1_promote_0 (call=161, rc=1) complete crmd[26524]: 2008/04/04_14:25:43 info: do_lrm_rsc_op: Performing op=ssh_2_stop_0 key=43:38:2987022a-c9a2-4803-a95c-d3a20c973d23) lrmd[26521]: 2008/04/04_14:25:43 info: rsc:ssh_2: stop lrmd[1601]: 2008/04/04_14:25:43 WARN: For LSB init script, no additional parameters are needed. crmd[26524]: 2008/04/04_14:25:43 info: do_lrm_rsc_op: Performing op=drbd0:1_notify_0 key=72:38:2987022a-c9a2-4803-a95c-d3a20c973d23) lrmd[26521]: 2008/04/04_14:25:43 info: rsc:drbd0:1: notify crmd[26524]: 2008/04/04_14:25:43 info: process_lrm_event: LRM operation ssh_2_monitor_120000 (call=158, rc=-2) Cancelled lrmd[26521]: 2008/04/04_14:25:43 info: RA output: (ssh_2:stop:stdout) Stopping OpenBSD Secure Shell server... lrmd[26521]: 2008/04/04_14:25:43 info: RA output: (ssh_2:stop:stdout) . crmd[26524]: 2008/04/04_14:25:43 info: process_lrm_event: LRM operation ssh_2_stop_0 (call=162, rc=0) complete crmd[26524]: 2008/04/04_14:25:43 info: process_lrm_event: LRM operation drbd0:1_notify_0 (call=163, rc=0) complete crmd[26524]: 2008/04/04_14:25:45 info: do_lrm_rsc_op: Performing op=IPaddr_192_168_2_150_stop_0 key=40:38:2987022a-c9a2-4803-a95c-d3a20c973d23) lrmd[26521]: 2008/04/04_14:25:45 info: rsc:IPaddr_192_168_2_150: stop crmd[26524]: 2008/04/04_14:25:45 info: do_lrm_rsc_op: Performing op=drbd0:1_demote_0 key=14:38:2987022a-c9a2-4803-a95c-d3a20c973d23) lrmd[26521]: 2008/04/04_14:25:45 info: rsc:drbd0:1: demote crmd[26524]: 2008/04/04_14:25:45 info: process_lrm_event: LRM operation IPaddr_192_168_2_150_monitor_5000 (call=156, rc=-2) Cancelled lrmd[26521]: 2008/04/04_14:25:45 info: RA output: (IPaddr_192_168_2_150:stop:stdout) In IP Stop lrmd[26521]: 2008/04/04_14:25:45 info: RA output: (IPaddr_192_168_2_150:stop:stderr) SIOCDELRT: No such process IPaddr[1634]: 2008/04/04_14:25:45 INFO: ifconfig eth0:0 down crmd[26524]: 2008/04/04_14:25:45 info: process_lrm_event: LRM operation IPaddr_192_168_2_150_stop_0 (call=164, rc=0) complete lrmd[26521]: 2008/04/04_14:25:45 info: RA output: (drbd0:1:demote:stdout) crmd[26524]: 2008/04/04_14:25:45 info: process_lrm_event: LRM operation drbd0:1_demote_0 (call=165, rc=0) complete crmd[26524]: 2008/04/04_14:25:46 info: do_lrm_rsc_op: Performing op=drbd0:1_notify_0 key=73:38:2987022a-c9a2-4803-a95c-d3a20c973d23) lrmd[26521]: 2008/04/04_14:25:46 info: rsc:drbd0:1: notify crm_master[1736]: 2008/04/04_14:25:46 info: Invoked: /usr/sbin/crm_master -v 75 -l reboot crmd[26524]: 2008/04/04_14:25:48 info: process_lrm_event: LRM operation drbd0:1_notify_0 (call=166, rc=0) complete lrmd[26521]: 2008/04/04_14:25:48 info: RA output: (drbd0:1:notify:stdout) No set matching id=master-23e734e2-b052-429a-b5ae-c3d31c4487a1 in status crmd[26524]: 2008/04/04_14:25:49 info: do_lrm_rsc_op: Performing op=drbd0:1_notify_0 key=67:39:2987022a-c9a2-4803-a95c-d3a20c973d23) lrmd[26521]: 2008/04/04_14:25:49 info: rsc:drbd0:1: notify crmd[26524]: 2008/04/04_14:25:49 info: process_lrm_event: LRM operation drbd0:1_notify_0 (call=167, rc=0) complete crmd[26524]: 2008/04/04_14:25:50 info: do_lrm_rsc_op: Performing op=drbd0:1_stop_0 key=3:39:2987022a-c9a2-4803-a95c-d3a20c973d23) lrmd[26521]: 2008/04/04_14:25:50 info: rsc:drbd0:1: stop crm_master[1818]: 2008/04/04_14:25:50 info: Invoked: /usr/sbin/crm_master -D lrmd[26521]: 2008/04/04_14:25:51 info: RA output: (drbd0:1:stop:stdout) lrmd[26521]: 2008/04/04_14:25:51 info: RA output: (drbd0:1:stop:stdout) crmd[26524]: 2008/04/04_14:25:51 info: process_lrm_event: LRM operation drbd0:1_stop_0 (call=168, rc=0) complete