待機系がフェールオーバーし、再起動を繰り返す (Linux-ha-jp) - Linux-HA Japan

広瀬　様
 
O.Nです。
早速、回答いただき、ありがとうございます。
説明不足で、申し訳ありません。

待機系のみを立ち上げた場合は、heartbeat、drbdも正常に起動し、
mount、httpd、PostgreSQLも起動できております。
 
DRBD領域には、DBのデータはあります。
稼働系、待機系をそれぞれ起動し、スプリットブレイン状態から
DRBDにて稼働系から待機系へデータ同期をおこうなことも可能です。
 
>> あと、haresourcesの中身が提示されたものと、ログの中が違う気がします。
サーバー、ドメイン名、メールアドレスはメール作成時に修正したため、一部修正漏れがあります。申し訳ありません。
その他のログは、原文のままになっております。
 
3．/etc/ha.d/haresourcesの抜粋は次の通りです。
SERVER1.domain drbddisk Filesystem::/dev/drbd0::/usr1::ext3 \
httpd postgresql 192.168.0.110/24 \
MailTo::test****@yahoo*****::server_FailOver
　
今回、スプリットブレイン後、DRBDにてデータを同期し、一旦、
両サーバを停止、再起動した場合、待機系サーバが再起動を繰り返します。
　ログから待機系は、稼働系を稼働していることも認識した後、数分後、
稼働系が正常稼働していないと判断しているように思います。
>> ha-debug側のログも取られているようですので、そちらの当該時間帯の
>> ログも提示していただけますでしょうか？
>>
はい、可能です。次の通りです。
 
heartbeat[3396]: 2013/06/03_13:19:40 debug: uid=hacluster, gid=<null>
heartbeat[3396]: 2013/06/03_13:19:40 debug: Creating authentication: uidptr=0x9913680 gidptr=0x0
heartbeat[3396]: 2013/06/03_13:19:40 debug: uid=<null>, gid=haclient
heartbeat[3396]: 2013/06/03_13:19:40 debug: Creating authentication: uidptr=0x0 gidptr=0x99136a0
heartbeat[3396]: 2013/06/03_13:19:40 debug: uid=root, gid=<null>
heartbeat[3396]: 2013/06/03_13:19:40 debug: Creating authentication: uidptr=0x99136c0 gidptr=0x0
heartbeat[3396]: 2013/06/03_13:19:40 debug: uid=<null>, gid=haclient
heartbeat[3396]: 2013/06/03_13:19:40 debug: Creating authentication: uidptr=0x0 gidptr=0x99136e0
heartbeat[3396]: 2013/06/03_13:19:40 debug: Beginning authentication parsing
heartbeat[3396]: 2013/06/03_13:19:40 debug: 16 max authentication methods
heartbeat[3396]: 2013/06/03_13:19:40 debug: Keyfile opened
heartbeat[3396]: 2013/06/03_13:19:40 debug: Keyfile perms OK
heartbeat[3396]: 2013/06/03_13:19:40 debug: 16 max authentication methods
heartbeat[3396]: 2013/06/03_13:19:40 debug: Found authentication method [crc]
heartbeat[3396]: 2013/06/03_13:19:40 info: AUTH: i=1: key = 0x991a168, auth=0x118c80, authname=crc
heartbeat[3396]: 2013/06/03_13:19:40 debug: Outbound signing method is 1
heartbeat[3396]: 2013/06/03_13:19:40 debug: Authentication parsing complete [1]
heartbeat[3396]: 2013/06/03_13:19:40 debug: add_option(cluster,linux-ha)
heartbeat[3396]: 2013/06/03_13:19:40 debug: add_option(hopfudge,1)
heartbeat[3396]: 2013/06/03_13:19:40 debug: add_option(baud,19200)
heartbeat[3396]: 2013/06/03_13:19:40 debug: add_option(hbgenmethod,file)
heartbeat[3396]: 2013/06/03_13:19:40 debug: add_option(realtime,true)
heartbeat[3396]: 2013/06/03_13:19:40 debug: add_option(msgfmt,classic)
heartbeat[3396]: 2013/06/03_13:19:40 debug: add_option(conn_logd_time,60)
heartbeat[3396]: 2013/06/03_13:19:40 debug: add_option(log_badpack,true)
heartbeat[3396]: 2013/06/03_13:19:40 debug: add_option(syslogmsgfmt,false)
heartbeat[3396]: 2013/06/03_13:19:40 debug: add_option(coredumps,true)
heartbeat[3396]: 2013/06/03_13:19:40 debug: add_option(crm,false)
heartbeat[3396]: 2013/06/03_13:19:40 info: Version 2 support: false
heartbeat[3396]: 2013/06/03_13:19:40 debug: add_option(autojoin,none)
heartbeat[3396]: 2013/06/03_13:19:40 debug: add_option(uuidfrom,file)
heartbeat[3396]: 2013/06/03_13:19:40 debug: add_option(compression,zlib)
heartbeat[3396]: 2013/06/03_13:19:40 debug: add_option(compression_threshold,2)
heartbeat[3396]: 2013/06/03_13:19:40 debug: add_option(traditional_compression,no)
heartbeat[3396]: 2013/06/03_13:19:40 debug: add_option(max_rexmit_delay,250)
heartbeat[3396]: 2013/06/03_13:19:40 debug: Setting max_rexmit_delay to 250 ms
heartbeat[3396]: 2013/06/03_13:19:40 debug: add_option(record_config_changes,on)
heartbeat[3396]: 2013/06/03_13:19:40 debug: add_option(record_pengine_inputs,on)
heartbeat[3396]: 2013/06/03_13:19:40 debug: add_option(enable_config_writes,on)
heartbeat[3396]: 2013/06/03_13:19:40 debug: add_option(memreserve,6500)
heartbeat[3396]: 2013/06/03_13:19:40 WARN: Logging daemon is disabled --enabling logging daemon is recommended
heartbeat[3396]: 2013/06/03_13:19:40 info: **************************
heartbeat[3396]: 2013/06/03_13:19:40 info: Configuration validated. Starting heartbeat 2.1.4
heartbeat[3396]: 2013/06/03_13:19:40 debug: HA configuration OK.  Heartbeat starting.
heartbeat[3398]: 2013/06/03_13:19:40 info: heartbeat: version 2.1.4
heartbeat[3398]: 2013/06/03_13:19:40 info: Heartbeat generation: 1369315411
heartbeat[3398]: 2013/06/03_13:19:40 debug: uuid is:dbb8c72b-1f87-4c24-a628-cd03e38df745
heartbeat[3398]: 2013/06/03_13:19:40 debug: FIFO process pid: 3419
heartbeat[3398]: 2013/06/03_13:19:40 debug: G_main_IPC_Channel_constructor(sock=5,5)
heartbeat[3398]: 2013/06/03_13:19:40 debug: opening ucast eth1 (UDP/IP unicast)
heartbeat[3398]: 2013/06/03_13:19:40 info: glib: ucast: write socket priority set to IPTOS_LOWDELAY on eth1
heartbeat[3398]: 2013/06/03_13:19:40 info: glib: ucast: bound send socket to device: eth1
heartbeat[3398]: 2013/06/03_13:19:40 info: glib: ucast: bound receive socket to device: eth1
heartbeat[3398]: 2013/06/03_13:19:40 info: glib: ucast: started on port 694 interface eth1 to 10.10.10.11
heartbeat[3398]: 2013/06/03_13:19:40 debug: write process pid: 3420
heartbeat[3398]: 2013/06/03_13:19:40 debug: read child process pid: 3421
heartbeat[3398]: 2013/06/03_13:19:40 debug: make_io_childpair: CREATED childpair wchan socket 10
heartbeat[3398]: 2013/06/03_13:19:40 debug: make_io_childpair: CREATED childpair rchan socket 12
heartbeat[3398]: 2013/06/03_13:19:40 debug: G_main_IPC_Channel_constructor(sock=10,10)
heartbeat[3398]: 2013/06/03_13:19:40 debug: G_main_IPC_Channel_constructor(sock=12,12)
heartbeat[3398]: 2013/06/03_13:19:40 debug: socket_destroy(ch=0x991dac0){
heartbeat[3398]: 2013/06/03_13:19:40 debug: socket_disconnect(sock=11, ch=0x991dac0){
heartbeat[3398]: 2013/06/03_13:19:40 debug: socket_disconnect: closing socket 11
heartbeat[3398]: 2013/06/03_13:19:40 debug: }/*socket_disconnect(sock=-1, ch=0x991dac0)*/
heartbeat[3398]: 2013/06/03_13:19:40 debug: }/*socket_destroy(ch=0x991dac0)*/
heartbeat[3398]: 2013/06/03_13:19:40 debug: socket_destroy(ch=0x991d728){
heartbeat[3398]: 2013/06/03_13:19:40 debug: socket_disconnect(sock=9, ch=0x991d728){
heartbeat[3398]: 2013/06/03_13:19:40 debug: socket_disconnect: closing socket 9
heartbeat[3398]: 2013/06/03_13:19:40 debug: }/*socket_disconnect(sock=-1, ch=0x991d728)*/
heartbeat[3398]: 2013/06/03_13:19:40 debug: }/*socket_destroy(ch=0x991d728)*/
heartbeat[3398]: 2013/06/03_13:19:40 debug: opening ping 192.168.0.1 (ping membership)
heartbeat[3398]: 2013/06/03_13:19:40 info: glib: ping heartbeat started.
heartbeat[3398]: 2013/06/03_13:19:40 debug: write process pid: 3422
heartbeat[3398]: 2013/06/03_13:19:40 debug: read child process pid: 3423
heartbeat[3398]: 2013/06/03_13:19:40 debug: make_io_childpair: CREATED childpair wchan socket 9
heartbeat[3398]: 2013/06/03_13:19:40 debug: make_io_childpair: CREATED childpair rchan socket 13
heartbeat[3398]: 2013/06/03_13:19:40 debug: G_main_IPC_Channel_constructor(sock=9,9)
heartbeat[3398]: 2013/06/03_13:19:40 debug: G_main_IPC_Channel_constructor(sock=13,13)
heartbeat[3398]: 2013/06/03_13:19:40 debug: socket_destroy(ch=0x9920450){
heartbeat[3398]: 2013/06/03_13:19:40 debug: socket_disconnect(sock=11, ch=0x9920450){
heartbeat[3398]: 2013/06/03_13:19:40 debug: socket_disconnect: closing socket 11
heartbeat[3398]: 2013/06/03_13:19:40 debug: }/*socket_disconnect(sock=-1, ch=0x9920450)*/
heartbeat[3398]: 2013/06/03_13:19:40 debug: }/*socket_destroy(ch=0x9920450)*/
heartbeat[3398]: 2013/06/03_13:19:40 debug: socket_destroy(ch=0x991d728){
heartbeat[3398]: 2013/06/03_13:19:40 debug: socket_disconnect(sock=8, ch=0x991d728){
heartbeat[3398]: 2013/06/03_13:19:40 debug: socket_disconnect: closing socket 8
heartbeat[3398]: 2013/06/03_13:19:40 debug: }/*socket_disconnect(sock=-1, ch=0x991d728)*/
heartbeat[3398]: 2013/06/03_13:19:40 debug: }/*socket_destroy(ch=0x991d728)*/
heartbeat[3398]: 2013/06/03_13:19:40 info: G_main_add_TriggerHandler: Added signal manual handler
heartbeat[3398]: 2013/06/03_13:19:40 info: G_main_add_TriggerHandler: Added signal manual handler
heartbeat[3398]: 2013/06/03_13:19:40 notice: Using watchdog device: /dev/watchdog
heartbeat[3398]: 2013/06/03_13:19:40 debug: Set watchdog timer to 61 seconds.
heartbeat[3398]: 2013/06/03_13:19:40 info: G_main_add_SignalHandler: Added signal handler for signal 17
heartbeat[3398]: 2013/06/03_13:19:40 debug: Limiting CPU: 42 CPU seconds every 60000 milliseconds
heartbeat[3398]: 2013/06/03_13:19:40 debug: pid 3398 locked in memory.
heartbeat[3398]: 2013/06/03_13:19:40 debug: Waiting for child processes to start
heartbeat[3398]: 2013/06/03_13:19:40 debug: PID 3398: Sending local status curnode = 807aaec status: init
heartbeat[3398]: 2013/06/03_13:19:40 debug: process_clustermsg: node [SEVER2.domain]
heartbeat[3398]: 2013/06/03_13:19:40 debug: process_clustermsg: node [SEVER2.domain]
heartbeat[3398]: 2013/06/03_13:19:40 debug: PID 3398: Sending local status curnode = 807aaec status: up
heartbeat[3398]: 2013/06/03_13:19:40 debug: process_clustermsg: node [SEVER2.domain]
heartbeat[3398]: 2013/06/03_13:19:40 info: Local status now set to: 'up'
heartbeat[3398]: 2013/06/03_13:19:40 debug: All your child process are belong to us
heartbeat[3398]: 2013/06/03_13:19:40 debug: PID 3398: Sending local status curnode = 807aaec status: up
heartbeat[3398]: 2013/06/03_13:19:40 debug: process_clustermsg: node [SEVER2.domain]
heartbeat[3398]: 2013/06/03_13:19:40 debug: Starting local status message @ 10000 ms intervals
heartbeat[3398]: 2013/06/03_13:19:40 debug: Forking temp process write_hostcachedata
heartbeat[3398]: 2013/06/03_13:19:40 info: Managed write_hostcachedata process 3431 exited with return code 0.
heartbeat[3419]: 2013/06/03_13:19:41 debug: pid 3419 locked in memory.
heartbeat[3419]: 2013/06/03_13:19:41 debug: Limiting CPU: 6 CPU seconds every 60000 milliseconds
heartbeat[3420]: 2013/06/03_13:19:41 debug: pid 3420 locked in memory.
heartbeat[3420]: 2013/06/03_13:19:41 debug: Limiting CPU: 24 CPU seconds every 60000 milliseconds
heartbeat[3421]: 2013/06/03_13:19:41 debug: pid 3421 locked in memory.
heartbeat[3421]: 2013/06/03_13:19:41 debug: Limiting CPU: 6 CPU seconds every 60000 milliseconds
heartbeat[3422]: 2013/06/03_13:19:41 debug: pid 3422 locked in memory.
heartbeat[3422]: 2013/06/03_13:19:41 debug: Limiting CPU: 24 CPU seconds every 60000 milliseconds
heartbeat[3422]: 2013/06/03_13:19:41 debug: Packet authenticated
heartbeat[3423]: 2013/06/03_13:19:41 debug: pid 3423 locked in memory.
heartbeat[3422]: 2013/06/03_13:19:41 debug: Packet authenticated
heartbeat[3423]: 2013/06/03_13:19:41 debug: Limiting CPU: 6 CPU seconds every 60000 milliseconds
heartbeat[3422]: 2013/06/03_13:19:41 debug: Packet authenticated
heartbeat[3422]: 2013/06/03_13:19:41 debug: Packet authenticated
heartbeat[3423]: 2013/06/03_13:19:41 debug: Packet authenticated
heartbeat[3423]: 2013/06/03_13:19:41 debug: Packet authenticated
heartbeat[3423]: 2013/06/03_13:19:41 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:19:41 debug: read_child_dispatch() {
heartbeat[3398]: 2013/06/03_13:19:41 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:19:41 debug: process_clustermsg: node [192.168.0.1]
heartbeat[3398]: 2013/06/03_13:19:41 info: Link 192.168.0.1:192.168.0.1 up.
heartbeat[3398]: 2013/06/03_13:19:41 debug: Queueing remote resource request (hook = 0x0x9913860) ifstat
heartbeat[3398]: 2013/06/03_13:19:41 debug: MSG: Dumping message with 4 fields
heartbeat[3398]: 2013/06/03_13:19:41 debug: MSG[0] : [t=ifstat]
heartbeat[3398]: 2013/06/03_13:19:41 debug: MSG[1] : [node=192.168.0.1]
heartbeat[3398]: 2013/06/03_13:19:41 debug: MSG[2] : [ifname=192.168.0.1]
heartbeat[3398]: 2013/06/03_13:19:41 debug: MSG[3] : [st=up]
heartbeat[3398]: 2013/06/03_13:19:41 debug: CreateInitialFilter: status
heartbeat[3398]: 2013/06/03_13:19:41 debug: CreateInitialFilter: ip-request-resp
heartbeat[3398]: 2013/06/03_13:19:41 debug: CreateInitialFilter: ip-request
heartbeat[3398]: 2013/06/03_13:19:41 debug: CreateInitialFilter: hb_takeover
heartbeat[3398]: 2013/06/03_13:19:41 debug: CreateInitialFilter: ask_resources
heartbeat[3398]: 2013/06/03_13:19:41 debug: FilterNotifications(ifstat) => 0
heartbeat[3398]: 2013/06/03_13:19:41 debug: ifstat: child process unneeded.
heartbeat[3398]: 2013/06/03_13:19:41 debug: MSG: Dumping message with 4 fields
heartbeat[3398]: 2013/06/03_13:19:41 debug: MSG[0] : [t=ifstat]
heartbeat[3398]: 2013/06/03_13:19:41 debug: MSG[1] : [node=192.168.0.1]
heartbeat[3398]: 2013/06/03_13:19:41 debug: MSG[2] : [ifname=192.168.0.1]
heartbeat[3398]: 2013/06/03_13:19:41 debug: MSG[3] : [st=up]
heartbeat[3398]: 2013/06/03_13:19:41 info: Status update for node 192.168.0.1: status ping
heartbeat[3398]: 2013/06/03_13:19:41 debug: Status seqno: 0 msgtime: 1370233180
heartbeat[3398]: 2013/06/03_13:19:41 debug: Queueing remote resource request (hook = 0x0x99138a0) NS_st
heartbeat[3398]: 2013/06/03_13:19:41 debug: MSG: Dumping message with 6 fields
heartbeat[3398]: 2013/06/03_13:19:41 debug: MSG[0] : [t=NS_st]
heartbeat[3398]: 2013/06/03_13:19:41 debug: MSG[1] : [st=ping]
heartbeat[3398]: 2013/06/03_13:19:41 debug: MSG[2] : [info=ping]
heartbeat[3398]: 2013/06/03_13:19:41 debug: MSG[3] : [src=192.168.0.1]
heartbeat[3398]: 2013/06/03_13:19:41 debug: MSG[4] : [ts=51ac195c]
heartbeat[3398]: 2013/06/03_13:19:41 debug: MSG[5] : [auth=1 cddd27f3]
heartbeat[3398]: 2013/06/03_13:19:41 debug: FilterNotifications(NS_st) => 0
heartbeat[3398]: 2013/06/03_13:19:41 debug: NS_st: child process unneeded.
heartbeat[3398]: 2013/06/03_13:19:41 debug: MSG: Dumping message with 6 fields
heartbeat[3398]: 2013/06/03_13:19:41 debug: MSG[0] : [t=NS_st]
heartbeat[3398]: 2013/06/03_13:19:41 debug: MSG[1] : [st=ping]
heartbeat[3398]: 2013/06/03_13:19:41 debug: MSG[2] : [info=ping]
heartbeat[3398]: 2013/06/03_13:19:41 debug: MSG[3] : [src=192.168.0.1]
heartbeat[3398]: 2013/06/03_13:19:41 debug: MSG[4] : [ts=51ac195c]
heartbeat[3398]: 2013/06/03_13:19:41 debug: MSG[5] : [auth=1 cddd27f3]
heartbeat[3398]: 2013/06/03_13:19:41 debug: }/*read_child_dispatch*/;
heartbeat[3398]: 2013/06/03_13:19:41 debug: read_child_dispatch() {
heartbeat[3398]: 2013/06/03_13:19:41 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:19:41 debug: process_clustermsg: node [SEVER1.domain]
heartbeat[3398]: 2013/06/03_13:19:41 info: Link SEVER1.domain:eth1 up.
heartbeat[3398]: 2013/06/03_13:19:41 debug: Queueing remote resource request (hook = 0x0x99138c0) ifstat
heartbeat[3398]: 2013/06/03_13:19:41 debug: MSG: Dumping message with 4 fields
heartbeat[3398]: 2013/06/03_13:19:41 debug: MSG[0] : [t=ifstat]
heartbeat[3398]: 2013/06/03_13:19:41 debug: MSG[1] : [node=SEVER1.domain]
heartbeat[3398]: 2013/06/03_13:19:41 debug: MSG[2] : [ifname=eth1]
heartbeat[3398]: 2013/06/03_13:19:41 debug: MSG[3] : [st=up]
heartbeat[3398]: 2013/06/03_13:19:41 debug: FilterNotifications(ifstat) => 0
heartbeat[3398]: 2013/06/03_13:19:41 debug: ifstat: child process unneeded.
heartbeat[3398]: 2013/06/03_13:19:41 debug: MSG: Dumping message with 4 fields
heartbeat[3398]: 2013/06/03_13:19:41 debug: MSG[0] : [t=ifstat]
heartbeat[3398]: 2013/06/03_13:19:41 debug: MSG[1] : [node=SEVER1.domain]
heartbeat[3398]: 2013/06/03_13:19:41 debug: MSG[2] : [ifname=eth1]
heartbeat[3398]: 2013/06/03_13:19:41 debug: MSG[3] : [st=up]
heartbeat[3398]: 2013/06/03_13:19:41 debug: sending reqnodes msg to node SEVER1.domain
heartbeat[3398]: 2013/06/03_13:19:41 debug: process_clustermsg: node [SEVER2.domain]
heartbeat[3398]: 2013/06/03_13:19:41 debug: }/*read_child_dispatch*/;
heartbeat[3398]: 2013/06/03_13:19:41 debug: read_child_dispatch() {
heartbeat[3422]: 2013/06/03_13:19:41 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:19:41 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:19:41 debug: process_clustermsg: node [192.168.0.1]
heartbeat[3398]: 2013/06/03_13:19:41 debug: }/*read_child_dispatch*/;
heartbeat[3398]: 2013/06/03_13:19:41 debug: read_child_dispatch() {
heartbeat[3398]: 2013/06/03_13:19:41 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:19:41 debug: process_clustermsg: node [192.168.0.1]
heartbeat[3398]: 2013/06/03_13:19:41 debug: }/*read_child_dispatch*/;
heartbeat[3398]: 2013/06/03_13:19:41 debug: Forking temp process write_hostcachedata
heartbeat[3398]: 2013/06/03_13:19:41 info: Managed write_hostcachedata process 3575 exited with return code 0.
heartbeat[3398]: 2013/06/03_13:19:42 debug: read_child_dispatch() {
heartbeat[3398]: 2013/06/03_13:19:42 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:19:42 debug: process_clustermsg: node [SEVER1.domain]
heartbeat[3398]: 2013/06/03_13:19:42 debug: Get a repnodes msg from SEVER1.domain
heartbeat[3398]: 2013/06/03_13:19:42 debug: nodelist received:SEVER1.domain SEVER2.domain 
heartbeat[3398]: 2013/06/03_13:19:42 info: Comm_now_up(): updating status to active
heartbeat[3398]: 2013/06/03_13:19:42 debug: PID 3398: Sending local status curnode = 807aaec status: active
heartbeat[3398]: 2013/06/03_13:19:42 debug: process_clustermsg: node [SEVER2.domain]
heartbeat[3398]: 2013/06/03_13:19:42 info: Local status now set to: 'active'
heartbeat[3398]: 2013/06/03_13:19:42 debug: Sending local starting msg: resourcestate = 0
heartbeat[3422]: 2013/06/03_13:19:42 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:19:42 debug: process_clustermsg: node [SEVER2.domain]
heartbeat[3423]: 2013/06/03_13:19:42 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:19:42 debug: hb_rsc_isstable: ResourceMgmt_child_count: 0, other_is_stable: 0, takeover_in_progress: 0, going_standby: 0, standby running(ms): 0, resourcestate: 0
heartbeat[3422]: 2013/06/03_13:19:42 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:19:42 info: Starting child client "/usr/lib/heartbeat/ipfail" (200,200)
heartbeat[3398]: 2013/06/03_13:19:43 info: Starting child client "/usr/local/sbin/check_active" (0,0)
heartbeat[3398]: 2013/06/03_13:19:43 debug: }/*read_child_dispatch*/;
heartbeat[3398]: 2013/06/03_13:19:43 WARN: G_CH_dispatch_int: Dispatch function for read child took too long to execute: 820 ms (> 50 ms) (GSource: 0x991df98)
heartbeat[3604]: 2013/06/03_13:19:43 info: Starting "/usr/lib/heartbeat/ipfail" as uid 200  gid 200 (pid 3604)
heartbeat[3398]: 2013/06/03_13:19:43 debug: read_child_dispatch() {
heartbeat[3398]: 2013/06/03_13:19:43 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:19:43 debug: process_clustermsg: node [192.168.0.1]
heartbeat[3398]: 2013/06/03_13:19:43 debug: }/*read_child_dispatch*/;
heartbeat[3605]: 2013/06/03_13:19:43 info: Starting "/usr/local/sbin/check_active" as uid 0  gid 0 (pid 3605)
heartbeat[3398]: 2013/06/03_13:19:43 debug: Forking temp process write_hostcachedata
heartbeat[3398]: 2013/06/03_13:19:43 debug: Forking temp process write_delcachedata
heartbeat[3398]: 2013/06/03_13:19:43 info: Managed write_hostcachedata process 3606 exited with return code 0.
heartbeat[3398]: 2013/06/03_13:19:43 info: Managed write_delcachedata process 3607 exited with return code 0.
heartbeat[3398]: 2013/06/03_13:19:43 WARN: G_SIG_dispatch: Dispatch function for SIGCHLD took too long to execute: 840 ms (> 30 ms) (GSource: 0x9920b20)
heartbeat[3398]: 2013/06/03_13:19:43 debug: read_child_dispatch() {
heartbeat[3398]: 2013/06/03_13:19:43 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:19:43 debug: process_clustermsg: node [SEVER1.domain]
heartbeat[3398]: 2013/06/03_13:19:43 info: AnnounceTakeover(local 0, foreign 1, reason 'T_RESOURCES' (0))
heartbeat[3398]: 2013/06/03_13:19:43 info: remote resource transition completed.
heartbeat[3398]: 2013/06/03_13:19:43 debug: Sending hold resources msg: none, stable=0 # <none>
heartbeat[3398]: 2013/06/03_13:19:43 debug: process_clustermsg: node [SEVER2.domain]
heartbeat[3398]: 2013/06/03_13:19:43 info: AnnounceTakeover(local 0, foreign 1, reason 'T_RESOURCES' (0))
heartbeat[3398]: 2013/06/03_13:19:43 info: STATE 1 => 3
heartbeat[3398]: 2013/06/03_13:19:43 debug: hb_rsc_isstable: ResourceMgmt_child_count: 0, other_is_stable: 1, takeover_in_progress: 0, going_standby: 0, standby running(ms): 0, resourcestate: 3
heartbeat[3398]: 2013/06/03_13:19:43 debug: Calling PerformAutoFailback()
heartbeat[3398]: 2013/06/03_13:19:43 info: other_holds_resources: 3
heartbeat[3422]: 2013/06/03_13:19:43 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:19:43 info: remote resource transition completed.
heartbeat[3398]: 2013/06/03_13:19:43 info: Local Resource acquisition completed. (none)
heartbeat[3398]: 2013/06/03_13:19:43 debug: Sending hold resources msg: none, stable=1 # <none>
heartbeat[3398]: 2013/06/03_13:19:43 debug: process_clustermsg: node [SEVER2.domain]
heartbeat[3398]: 2013/06/03_13:19:43 debug: hb_rsc_isstable: ResourceMgmt_child_count: 0, other_is_stable: 1, takeover_in_progress: 0, going_standby: 0, standby running(ms): 0, resourcestate: 3
heartbeat[3398]: 2013/06/03_13:19:43 debug: Calling PerformAutoFailback()
heartbeat[3398]: 2013/06/03_13:19:43 info: AnnounceTakeover(local 1, foreign 1, reason 'T_RESOURCES(them)' (0))
heartbeat[3398]: 2013/06/03_13:19:43 info: Initial resource acquisition complete (T_RESOURCES(them))
heartbeat[3422]: 2013/06/03_13:19:43 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:19:43 info: STATE 3 => 4
heartbeat[3398]: 2013/06/03_13:19:43 debug: hb_rsc_isstable: ResourceMgmt_child_count: 0, other_is_stable: 1, takeover_in_progress: 0, going_standby: 0, standby running(ms): 0, resourcestate: 4
heartbeat[3398]: 2013/06/03_13:19:43 debug: }/*read_child_dispatch*/;
heartbeat[3398]: 2013/06/03_13:19:43 WARN: G_SIG_dispatch: Dispatch function for SIGCHLD was delayed 850 ms (> 100 ms) before being called (GSource: 0x9920b20)
heartbeat[3398]: 2013/06/03_13:19:43 info: G_SIG_dispatch: started at 429409420 should have started at 429409335
ipfail[3604]: 2013/06/03_13:19:44 debug: PID=3604
ipfail[3604]: 2013/06/03_13:19:44 debug: Signing in with heartbeat
heartbeat[3398]: 2013/06/03_13:19:44 debug: read_child_dispatch() {
heartbeat[3398]: 2013/06/03_13:19:44 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:19:44 debug: process_clustermsg: node [SEVER1.domain]
heartbeat[3398]: 2013/06/03_13:19:44 info: other_holds_resources: 3
heartbeat[3398]: 2013/06/03_13:19:44 debug: hb_rsc_isstable: ResourceMgmt_child_count: 0, other_is_stable: 1, takeover_in_progress: 0, going_standby: 0, standby running(ms): 0, resourcestate: 4
heartbeat[3398]: 2013/06/03_13:19:44 debug: }/*read_child_dispatch*/;
heartbeat[3398]: 2013/06/03_13:19:44 debug: APIregistration_dispatch() {
heartbeat[3398]: 2013/06/03_13:19:44 debug: process_registerevent() {
heartbeat[3398]: 2013/06/03_13:19:44 debug: G_main_IPC_Channel_constructor(sock=14,14)
heartbeat[3398]: 2013/06/03_13:19:44 debug: client->gsource = 0x992b5e0
heartbeat[3398]: 2013/06/03_13:19:44 debug: }/*process_registerevent*/;
heartbeat[3398]: 2013/06/03_13:19:44 debug: }/*APIregistration_dispatch*/;
heartbeat[3398]: 2013/06/03_13:19:44 WARN: G_WC_dispatch: Dispatch function for client registration took too long to execute: 640 ms (> 20 ms) (GSource: 0x992d878)
heartbeat[3398]: 2013/06/03_13:19:44 debug: APIclients_input_dispatch() {
heartbeat[3398]: 2013/06/03_13:19:44 debug: ProcessAnAPIRequest() {
heartbeat[3398]: 2013/06/03_13:19:44 debug: api_process_registration_msg(hbapi-req, 3604, ipfail)
heartbeat[3398]: 2013/06/03_13:19:44 debug: Checking client authorization for client ipfail (200:200)
heartbeat[3398]: 2013/06/03_13:19:44 debug: process_clustermsg: node [SEVER2.domain]
heartbeat[3398]: 2013/06/03_13:19:44 debug: Queueing remote resource request (hook = 0x0x9913940) hbapi-clstat
heartbeat[3398]: 2013/06/03_13:19:44 debug: MSG: Dumping message with 14 fields
heartbeat[3398]: 2013/06/03_13:19:44 debug: MSG[0] : [t=hbapi-clstat]
heartbeat[3398]: 2013/06/03_13:19:44 debug: MSG[1] : [st=join]
heartbeat[3398]: 2013/06/03_13:19:44 debug: MSG[2] : [from_id=ipfail]
heartbeat[3398]: 2013/06/03_13:19:44 debug: MSG[3] : [to_id=ipfail]
heartbeat[3398]: 2013/06/03_13:19:44 debug: MSG[4] : [src=SEVER2.domain]
heartbeat[3398]: 2013/06/03_13:19:44 debug: MSG[5] : [info=signon]
heartbeat[3398]: 2013/06/03_13:19:44 debug: MSG[6] : [client_gen=0]
heartbeat[3398]: 2013/06/03_13:19:44 debug: MSG[7] : [(1)srcuuid=0x99315c8(36 27)]
heartbeat[3398]: 2013/06/03_13:19:44 debug: MSG[8] : [seq=9]
heartbeat[3398]: 2013/06/03_13:19:44 debug: MSG[9] : [hg=519e1853]
heartbeat[3398]: 2013/06/03_13:19:44 debug: MSG[10] : [ts=51ac1960]
heartbeat[3398]: 2013/06/03_13:19:44 debug: MSG[11] : [ld=1.26 0.36 0.12 5/145 3614]
heartbeat[3398]: 2013/06/03_13:19:44 debug: MSG[12] : [ttl=4]
heartbeat[3398]: 2013/06/03_13:19:44 debug: MSG[13] : [auth=1 1ed8a3f7]
heartbeat[3398]: 2013/06/03_13:19:44 debug: FilterNotifications(hbapi-clstat) => 0
heartbeat[3398]: 2013/06/03_13:19:44 debug: hbapi-clstat: child process unneeded.
heartbeat[3398]: 2013/06/03_13:19:44 debug: MSG: Dumping message with 14 fields
heartbeat[3398]: 2013/06/03_13:19:44 debug: MSG[0] : [t=hbapi-clstat]
heartbeat[3398]: 2013/06/03_13:19:44 debug: MSG[1] : [st=join]
heartbeat[3398]: 2013/06/03_13:19:44 debug: MSG[2] : [from_id=ipfail]
heartbeat[3398]: 2013/06/03_13:19:44 debug: MSG[3] : [to_id=ipfail]
heartbeat[3398]: 2013/06/03_13:19:44 debug: MSG[4] : [src=SEVER2.domain]
heartbeat[3398]: 2013/06/03_13:19:44 debug: MSG[5] : [info=signon]
heartbeat[3398]: 2013/06/03_13:19:44 debug: MSG[6] : [client_gen=0]
heartbeat[3398]: 2013/06/03_13:19:44 debug: MSG[7] : [(1)srcuuid=0x99315c8(36 27)]
heartbeat[3398]: 2013/06/03_13:19:44 debug: MSG[8] : [seq=9]
heartbeat[3398]: 2013/06/03_13:19:44 debug: MSG[9] : [hg=519e1853]
heartbeat[3398]: 2013/06/03_13:19:44 debug: MSG[10] : [ts=51ac1960]
heartbeat[3398]: 2013/06/03_13:19:44 debug: MSG[11] : [ld=1.26 0.36 0.12 5/145 3614]
heartbeat[3398]: 2013/06/03_13:19:44 debug: MSG[12] : [ttl=4]
heartbeat[3398]: 2013/06/03_13:19:44 debug: MSG[13] : [auth=1 1ed8a3f7]
heartbeat[3398]: 2013/06/03_13:19:44 debug: create_seq_snapshot_table:no missing packets found for node SEVER1.domain
heartbeat[3398]: 2013/06/03_13:19:44 debug: create_seq_snapshot_table:no missing packets found for node SEVER2.domain
heartbeat[3422]: 2013/06/03_13:19:44 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:19:44 debug: Signing on API client 3604 (ipfail)
heartbeat[3398]: 2013/06/03_13:19:44 debug:  return TRUE;
heartbeat[3398]: 2013/06/03_13:19:44 debug: }/*ProcessAnAPIRequest*/;
heartbeat[3398]: 2013/06/03_13:19:44 debug: return 1;
heartbeat[3398]: 2013/06/03_13:19:44 debug: }/*APIclients_input_dispatch*/;
heartbeat[3398]: 2013/06/03_13:19:44 debug: APIclients_input_dispatch() {
heartbeat[3398]: 2013/06/03_13:19:44 debug: ProcessAnAPIRequest() {
heartbeat[3398]: 2013/06/03_13:19:44 debug: hb_rsc_isstable: ResourceMgmt_child_count: 0, other_is_stable: 1, takeover_in_progress: 0, going_standby: 0, standby running(ms): 0, resourcestate: 4
heartbeat[3398]: 2013/06/03_13:19:44 debug:  return TRUE;
heartbeat[3398]: 2013/06/03_13:19:44 debug: }/*ProcessAnAPIRequest*/;
ipfail[3604]: 2013/06/03_13:19:44 debug: [We are SEVER2.domain]
heartbeat[3398]: 2013/06/03_13:19:44 debug: return 1;
heartbeat[3398]: 2013/06/03_13:19:44 debug: }/*APIclients_input_dispatch*/;
heartbeat[3398]: 2013/06/03_13:19:44 debug: APIclients_input_dispatch() {
heartbeat[3398]: 2013/06/03_13:19:44 debug: ProcessAnAPIRequest() {
heartbeat[3398]: 2013/06/03_13:19:44 debug:  return TRUE;
heartbeat[3398]: 2013/06/03_13:19:44 debug: }/*ProcessAnAPIRequest*/;
heartbeat[3398]: 2013/06/03_13:19:44 debug: return 1;
heartbeat[3398]: 2013/06/03_13:19:44 debug: }/*APIclients_input_dispatch*/;
ipfail[3604]: 2013/06/03_13:19:44 debug: auto_failback -> 0 (off)
heartbeat[3398]: 2013/06/03_13:19:45 debug: APIclients_input_dispatch() {
heartbeat[3398]: 2013/06/03_13:19:45 debug: ProcessAnAPIRequest() {
heartbeat[3398]: 2013/06/03_13:19:45 debug:  return TRUE;
heartbeat[3398]: 2013/06/03_13:19:45 debug: }/*ProcessAnAPIRequest*/;
heartbeat[3398]: 2013/06/03_13:19:45 debug: return 1;
heartbeat[3398]: 2013/06/03_13:19:45 debug: }/*APIclients_input_dispatch*/;
ipfail[3604]: 2013/06/03_13:19:45 debug: Setting message filter mode
heartbeat[3398]: 2013/06/03_13:19:45 debug: PID 3398: Sending local status curnode = 807aaec status: active
heartbeat[3398]: 2013/06/03_13:19:45 debug: process_clustermsg: node [SEVER2.domain]
heartbeat[3398]: 2013/06/03_13:19:45 debug: APIclients_input_dispatch() {
heartbeat[3422]: 2013/06/03_13:19:45 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:19:45 debug: ProcessAnAPIRequest() {
heartbeat[3398]: 2013/06/03_13:19:45 debug:  return TRUE;
ipfail[3604]: 2013/06/03_13:19:45 debug: Starting node walk
heartbeat[3398]: 2013/06/03_13:19:45 debug: }/*ProcessAnAPIRequest*/;
heartbeat[3398]: 2013/06/03_13:19:45 debug: return 1;
heartbeat[3398]: 2013/06/03_13:19:45 debug: }/*APIclients_input_dispatch*/;
heartbeat[3398]: 2013/06/03_13:19:45 debug: APIclients_input_dispatch() {
heartbeat[3398]: 2013/06/03_13:19:45 debug: ProcessAnAPIRequest() {
heartbeat[3398]: 2013/06/03_13:19:45 debug:  return TRUE;
heartbeat[3398]: 2013/06/03_13:19:45 debug: }/*ProcessAnAPIRequest*/;
heartbeat[3398]: 2013/06/03_13:19:45 debug: return 1;
heartbeat[3398]: 2013/06/03_13:19:45 debug: }/*APIclients_input_dispatch*/;
heartbeat[3398]: 2013/06/03_13:19:45 debug: APIclients_input_dispatch() {
heartbeat[3398]: 2013/06/03_13:19:45 debug: ProcessAnAPIRequest() {
heartbeat[3398]: 2013/06/03_13:19:45 debug:  return TRUE;
ipfail[3604]: 2013/06/03_13:19:45 debug: Cluster node: 192.168.0.1: status: ping
heartbeat[3398]: 2013/06/03_13:19:45 debug: }/*ProcessAnAPIRequest*/;
heartbeat[3398]: 2013/06/03_13:19:45 debug: return 1;
heartbeat[3398]: 2013/06/03_13:19:45 debug: }/*APIclients_input_dispatch*/;
heartbeat[3398]: 2013/06/03_13:19:45 debug: APIclients_input_dispatch() {
heartbeat[3398]: 2013/06/03_13:19:45 debug: ProcessAnAPIRequest() {
heartbeat[3398]: 2013/06/03_13:19:45 debug:  return TRUE;
heartbeat[3398]: 2013/06/03_13:19:45 debug: }/*ProcessAnAPIRequest*/;
heartbeat[3398]: 2013/06/03_13:19:45 debug: return 1;
heartbeat[3398]: 2013/06/03_13:19:45 debug: }/*APIclients_input_dispatch*/;
heartbeat[3398]: 2013/06/03_13:19:45 debug: APIclients_input_dispatch() {
heartbeat[3398]: 2013/06/03_13:19:45 debug: ProcessAnAPIRequest() {
heartbeat[3398]: 2013/06/03_13:19:45 debug:  return TRUE;
ipfail[3604]: 2013/06/03_13:19:45 debug: Cluster node: SEVER2.domain: status: active
heartbeat[3398]: 2013/06/03_13:19:45 debug: }/*ProcessAnAPIRequest*/;
heartbeat[3398]: 2013/06/03_13:19:45 debug: return 1;
heartbeat[3398]: 2013/06/03_13:19:45 debug: }/*APIclients_input_dispatch*/;
heartbeat[3398]: 2013/06/03_13:19:45 debug: APIclients_input_dispatch() {
heartbeat[3398]: 2013/06/03_13:19:45 debug: ProcessAnAPIRequest() {
heartbeat[3398]: 2013/06/03_13:19:45 debug:  return TRUE;
heartbeat[3398]: 2013/06/03_13:19:45 debug: }/*ProcessAnAPIRequest*/;
heartbeat[3398]: 2013/06/03_13:19:45 debug: return 1;
heartbeat[3398]: 2013/06/03_13:19:45 debug: }/*APIclients_input_dispatch*/;
heartbeat[3398]: 2013/06/03_13:19:45 debug: APIclients_input_dispatch() {
heartbeat[3398]: 2013/06/03_13:19:45 debug: ProcessAnAPIRequest() {
heartbeat[3398]: 2013/06/03_13:19:45 debug:  return TRUE;
heartbeat[3398]: 2013/06/03_13:19:45 debug: }/*ProcessAnAPIRequest*/;
ipfail[3604]: 2013/06/03_13:19:45 debug: Cluster node: SEVER1.domain: status: init
heartbeat[3398]: 2013/06/03_13:19:45 debug: return 1;
heartbeat[3423]: 2013/06/03_13:19:45 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:19:45 debug: }/*APIclients_input_dispatch*/;
heartbeat[3398]: 2013/06/03_13:19:45 debug: read_child_dispatch() {
heartbeat[3398]: 2013/06/03_13:19:45 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:19:45 debug: process_clustermsg: node [192.168.0.1]
heartbeat[3398]: 2013/06/03_13:19:45 debug: }/*read_child_dispatch*/;
heartbeat[3398]: 2013/06/03_13:19:45 debug: APIclients_input_dispatch() {
heartbeat[3398]: 2013/06/03_13:19:45 debug: ProcessAnAPIRequest() {
heartbeat[3398]: 2013/06/03_13:19:45 debug:  return TRUE;
ipfail[3604]: 2013/06/03_13:19:45 debug: [They are SEVER1.domain]
heartbeat[3398]: 2013/06/03_13:19:45 debug: }/*ProcessAnAPIRequest*/;
heartbeat[3398]: 2013/06/03_13:19:45 debug: return 1;
ipfail[3604]: 2013/06/03_13:19:45 debug: Setting message signal
heartbeat[3398]: 2013/06/03_13:19:45 debug: }/*APIclients_input_dispatch*/;
heartbeat[3398]: 2013/06/03_13:19:45 debug: APIclients_input_dispatch() {
heartbeat[3398]: 2013/06/03_13:19:45 debug: ProcessAnAPIRequest() {
heartbeat[3398]: 2013/06/03_13:19:45 debug:  return TRUE;
ipfail[3604]: 2013/06/03_13:19:45 debug: Waiting for messages...
heartbeat[3398]: 2013/06/03_13:19:45 debug: }/*ProcessAnAPIRequest*/;
heartbeat[3398]: 2013/06/03_13:19:45 debug: return 1;
ipfail[3604]: 2013/06/03_13:19:45 debug: G_main_IPC_Channel_constructor(sock=4,4)
heartbeat[3398]: 2013/06/03_13:19:45 debug: }/*APIclients_input_dispatch*/;
heartbeat[3398]: 2013/06/03_13:19:46 debug: read_child_dispatch() {
heartbeat[3398]: 2013/06/03_13:19:46 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:19:46 debug: process_clustermsg: node [SEVER1.domain]
heartbeat[3398]: 2013/06/03_13:19:46 debug: process_clustermsg: node [SEVER2.domain]
heartbeat[3398]: 2013/06/03_13:19:46 info: Status update for node SEVER1.domain: status active
heartbeat[3398]: 2013/06/03_13:19:46 debug: Status seqno: 631 msgtime: 1370233206
heartbeat[3422]: 2013/06/03_13:19:46 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:19:46 debug: Queueing remote resource request (hook = 0x0x9913960) status
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG: Dumping message with 12 fields
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[0] : [t=status]
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[1] : [st=active]
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[2] : [dt=ea60]
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[3] : [protocol=1]
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[4] : [src=SEVER1.domain]
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[5] : [(1)srcuuid=0x9931a28(36 27)]
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[6] : [seq=277]
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[7] : [hg=518ca998]
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[8] : [ts=51ac1976]
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[9] : [ld=0.43 0.65 0.42 1/303 10821]
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[10] : [ttl=4]
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[11] : [auth=1 61167c5e]
heartbeat[3398]: 2013/06/03_13:19:46 debug: FilterNotifications(status) => 1
heartbeat[3398]: 2013/06/03_13:19:46 debug: StartNextRemoteRscReq() - calling hook
heartbeat[3398]: 2013/06/03_13:19:46 debug: PerformQueuedNotifyWorld() msg follows
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG: Dumping message with 12 fields
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[0] : [t=status]
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[1] : [st=active]
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[2] : [dt=ea60]
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[3] : [protocol=1]
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[4] : [src=SEVER1.domain]
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[5] : [(1)srcuuid=0x9932828(36 27)]
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[6] : [seq=277]
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[7] : [hg=518ca998]
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[8] : [ts=51ac1976]
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[9] : [ld=0.43 0.65 0.42 1/303 10821]
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[10] : [ttl=4]
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[11] : [auth=1 61167c5e]
heartbeat[3398]: 2013/06/03_13:19:46 debug: FilterNotifications(status) => 1
heartbeat[3398]: 2013/06/03_13:19:46 debug: notify_world: invoking harc: OLD status: active
heartbeat[3398]: 2013/06/03_13:19:46 debug: Process [status] started pid 3682
heartbeat[3398]: 2013/06/03_13:19:46 debug: Starting notify process [status]
heartbeat[3398]: 2013/06/03_13:19:46 debug: }/*read_child_dispatch*/;
ipfail[3604]: 2013/06/03_13:19:46 info: Status update: Node SEVER1.domain now has status active
heartbeat[3398]: 2013/06/03_13:19:46 debug: read_child_dispatch() {
heartbeat[3398]: 2013/06/03_13:19:46 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:19:46 debug: process_clustermsg: node [SEVER1.domain]
heartbeat[3398]: 2013/06/03_13:19:46 debug: Queueing remote resource request (hook = 0x0x9913980) num_ping_nodes
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG: Dumping message with 15 fields
ipfail[3604]: 2013/06/03_13:19:46 debug: Got asked for num_ping.
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[0] : [t=num_ping_nodes]
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[1] : [src=SEVER1.domain]
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[2] : [num_ping=1]
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[3] : [dest=SEVER2.domain]
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[4] : [from_id=ipfail]
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[5] : [to_id=ipfail]
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[6] : [client_gen=0]
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[7] : [(1)destuuid=0x9932148(37 28)]
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[8] : [(1)srcuuid=0x99320d8(36 27)]
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[9] : [seq=278]
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[10] : [hg=518ca998]
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[11] : [ts=51ac1976]
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[12] : [ld=0.43 0.65 0.42 1/303 10821]
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[13] : [ttl=4]
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[14] : [auth=1 3252806e]
heartbeat[3398]: 2013/06/03_13:19:46 debug: FilterNotifications(num_ping_nodes) => 0
heartbeat[3398]: 2013/06/03_13:19:46 debug: num_ping_nodes: child process unneeded.
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG: Dumping message with 15 fields
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[0] : [t=num_ping_nodes]
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[1] : [src=SEVER1.domain]
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[2] : [num_ping=1]
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[3] : [dest=SEVER2.domain]
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[4] : [from_id=ipfail]
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[5] : [to_id=ipfail]
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[6] : [client_gen=0]
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[7] : [(1)destuuid=0x9932148(37 28)]
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[8] : [(1)srcuuid=0x99320d8(36 27)]
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[9] : [seq=278]
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[10] : [hg=518ca998]
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[11] : [ts=51ac1976]
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[12] : [ld=0.43 0.65 0.42 1/303 10821]
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[13] : [ttl=4]
heartbeat[3682]: 2013/06/03_13:19:46 debug: notify_world: setting SIGCHLD Handler to SIG_DFL
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[14] : [auth=1 3252806e]
heartbeat[3398]: 2013/06/03_13:19:46 debug: }/*read_child_dispatch*/;
heartbeat[3682]: 2013/06/03_13:19:46 debug: notify_world: Running harc status
heartbeat[3398]: 2013/06/03_13:19:46 debug: APIclients_input_dispatch() {
heartbeat[3398]: 2013/06/03_13:19:46 debug: ProcessAnAPIRequest() {
heartbeat[3398]: 2013/06/03_13:19:46 debug:  return TRUE;
heartbeat[3398]: 2013/06/03_13:19:46 debug: }/*ProcessAnAPIRequest*/;
heartbeat[3398]: 2013/06/03_13:19:46 debug: return 1;
heartbeat[3398]: 2013/06/03_13:19:46 debug: }/*APIclients_input_dispatch*/;
heartbeat[3398]: 2013/06/03_13:19:46 debug: APIclients_input_dispatch() {
heartbeat[3398]: 2013/06/03_13:19:46 debug: ProcessAnAPIRequest() {
heartbeat[3398]: 2013/06/03_13:19:46 debug:  return TRUE;
ipfail[3604]: 2013/06/03_13:19:46 debug: Found ping node 192.168.0.1!
heartbeat[3398]: 2013/06/03_13:19:46 debug: }/*ProcessAnAPIRequest*/;
heartbeat[3398]: 2013/06/03_13:19:46 debug: return 1;
heartbeat[3398]: 2013/06/03_13:19:46 debug: }/*APIclients_input_dispatch*/;
heartbeat[3398]: 2013/06/03_13:19:46 debug: APIclients_input_dispatch() {
heartbeat[3398]: 2013/06/03_13:19:46 debug: ProcessAnAPIRequest() {
heartbeat[3398]: 2013/06/03_13:19:46 debug:  return TRUE;
heartbeat[3398]: 2013/06/03_13:19:46 debug: }/*ProcessAnAPIRequest*/;
heartbeat[3398]: 2013/06/03_13:19:46 debug: return 1;
heartbeat[3398]: 2013/06/03_13:19:46 debug: }/*APIclients_input_dispatch*/;
heartbeat[3398]: 2013/06/03_13:19:46 debug: APIclients_input_dispatch() {
heartbeat[3398]: 2013/06/03_13:19:46 debug: ProcessAnAPIRequest() {
heartbeat[3398]: 2013/06/03_13:19:46 debug:  return TRUE;
heartbeat[3398]: 2013/06/03_13:19:46 debug: }/*ProcessAnAPIRequest*/;
heartbeat[3398]: 2013/06/03_13:19:46 debug: return 1;
ipfail[3604]: 2013/06/03_13:19:46 info: Ping node count is balanced.
heartbeat[3398]: 2013/06/03_13:19:46 debug: }/*APIclients_input_dispatch*/;
ipfail[3604]: 2013/06/03_13:19:46 debug: Abort message sent.
heartbeat[3398]: 2013/06/03_13:19:46 debug: APIclients_input_dispatch() {
heartbeat[3398]: 2013/06/03_13:19:46 debug: ProcessAnAPIRequest() {
heartbeat[3398]: 2013/06/03_13:19:46 debug: Sending API message to cluster...
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG: Dumping message with 5 fields
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[0] : [t=abort_giveup]
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[1] : [src=SEVER2.domain]
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[2] : [dest=SEVER1.domain]
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[3] : [from_id=ipfail]
heartbeat[3398]: 2013/06/03_13:19:46 debug: MSG[4] : [to_id=ipfail]
heartbeat[3398]: 2013/06/03_13:19:46 debug: process_clustermsg: node [SEVER2.domain]
heartbeat[3398]: 2013/06/03_13:19:46 debug: process_clustermsg: node [SEVER2.domain]
heartbeat[3398]: 2013/06/03_13:19:46 debug:  return TRUE;
heartbeat[3398]: 2013/06/03_13:19:46 debug: }/*ProcessAnAPIRequest*/;
heartbeat[3398]: 2013/06/03_13:19:46 debug: return 1;
heartbeat[3422]: 2013/06/03_13:19:46 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:19:46 debug: }/*APIclients_input_dispatch*/;
heartbeat[3422]: 2013/06/03_13:19:46 debug: Packet authenticated
harc[3682]: 2013/06/03_13:19:46 info: Running /etc/ha.d/rc.d/status status
heartbeat[3398]: 2013/06/03_13:19:46 info: Managed status process 3682 exited with return code 0.
heartbeat[3398]: 2013/06/03_13:19:46 debug: RscMgmtProc 'status' exited code 0
heartbeat[3398]: 2013/06/03_13:19:50 debug: hb_send_local_status() {
heartbeat[3398]: 2013/06/03_13:19:50 debug: PID 3398: Sending local status curnode = 807aaec status: active
heartbeat[3398]: 2013/06/03_13:19:50 debug: process_clustermsg: node [SEVER2.domain]
heartbeat[3398]: 2013/06/03_13:19:50 debug: }/*hb_send_local_status*/;
heartbeat[3422]: 2013/06/03_13:19:50 debug: Packet authenticated
heartbeat[3423]: 2013/06/03_13:19:50 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:19:50 debug: read_child_dispatch() {
heartbeat[3398]: 2013/06/03_13:19:50 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:19:50 debug: process_clustermsg: node [192.168.0.1]
heartbeat[3398]: 2013/06/03_13:19:50 debug: }/*read_child_dispatch*/;
heartbeat[3398]: 2013/06/03_13:19:56 debug: read_child_dispatch() {
heartbeat[3398]: 2013/06/03_13:19:56 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:19:56 debug: process_clustermsg: node [SEVER1.domain]
heartbeat[3398]: 2013/06/03_13:19:56 debug: }/*read_child_dispatch*/;
heartbeat[3398]: 2013/06/03_13:19:56 debug: read_child_dispatch() {
heartbeat[3398]: 2013/06/03_13:19:56 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:19:56 debug: process_clustermsg: node [SEVER1.domain]
heartbeat[3398]: 2013/06/03_13:19:56 debug: }/*read_child_dispatch*/;
heartbeat[3398]: 2013/06/03_13:20:00 debug: hb_send_local_status() {
heartbeat[3398]: 2013/06/03_13:20:00 debug: PID 3398: Sending local status curnode = 807aaec status: active
heartbeat[3398]: 2013/06/03_13:20:00 debug: process_clustermsg: node [SEVER2.domain]
heartbeat[3398]: 2013/06/03_13:20:00 debug: }/*hb_send_local_status*/;
heartbeat[3422]: 2013/06/03_13:20:00 debug: Packet authenticated
heartbeat[3423]: 2013/06/03_13:20:00 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:20:00 debug: read_child_dispatch() {
heartbeat[3398]: 2013/06/03_13:20:00 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:20:00 debug: process_clustermsg: node [SEVER1.domain]
heartbeat[3398]: 2013/06/03_13:20:00 debug: }/*read_child_dispatch*/;
heartbeat[3398]: 2013/06/03_13:20:00 debug: read_child_dispatch() {
heartbeat[3398]: 2013/06/03_13:20:00 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:20:00 debug: process_clustermsg: node [192.168.0.1]
heartbeat[3398]: 2013/06/03_13:20:00 debug: }/*read_child_dispatch*/;
heartbeat[3398]: 2013/06/03_13:20:06 debug: read_child_dispatch() {
heartbeat[3398]: 2013/06/03_13:20:06 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:20:06 debug: process_clustermsg: node [SEVER1.domain]
heartbeat[3398]: 2013/06/03_13:20:06 debug: }/*read_child_dispatch*/;
heartbeat[3398]: 2013/06/03_13:20:10 debug: hb_send_local_status() {
heartbeat[3398]: 2013/06/03_13:20:10 debug: PID 3398: Sending local status curnode = 807aaec status: active
heartbeat[3398]: 2013/06/03_13:20:10 debug: process_clustermsg: node [SEVER2.domain]
heartbeat[3398]: 2013/06/03_13:20:10 debug: }/*hb_send_local_status*/;
heartbeat[3422]: 2013/06/03_13:20:10 debug: Packet authenticated
heartbeat[3423]: 2013/06/03_13:20:10 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:20:10 debug: read_child_dispatch() {
heartbeat[3398]: 2013/06/03_13:20:10 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:20:10 debug: process_clustermsg: node [192.168.0.1]
heartbeat[3398]: 2013/06/03_13:20:10 debug: }/*read_child_dispatch*/;
heartbeat[3398]: 2013/06/03_13:20:16 debug: read_child_dispatch() {
heartbeat[3398]: 2013/06/03_13:20:16 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:20:16 debug: process_clustermsg: node [SEVER1.domain]
heartbeat[3398]: 2013/06/03_13:20:16 debug: }/*read_child_dispatch*/;
heartbeat[3398]: 2013/06/03_13:20:20 debug: hb_send_local_status() {
heartbeat[3398]: 2013/06/03_13:20:20 debug: PID 3398: Sending local status curnode = 807aaec status: active
heartbeat[3398]: 2013/06/03_13:20:20 debug: process_clustermsg: node [SEVER2.domain]
heartbeat[3398]: 2013/06/03_13:20:20 debug: }/*hb_send_local_status*/;
heartbeat[3422]: 2013/06/03_13:20:20 debug: Packet authenticated
heartbeat[3423]: 2013/06/03_13:20:20 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:20:20 debug: read_child_dispatch() {
heartbeat[3398]: 2013/06/03_13:20:20 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:20:20 debug: process_clustermsg: node [192.168.0.1]
heartbeat[3398]: 2013/06/03_13:20:20 debug: }/*read_child_dispatch*/;
heartbeat[3398]: 2013/06/03_13:20:26 debug: read_child_dispatch() {
heartbeat[3398]: 2013/06/03_13:20:26 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:20:26 debug: process_clustermsg: node [SEVER1.domain]
heartbeat[3398]: 2013/06/03_13:20:26 debug: }/*read_child_dispatch*/;
heartbeat[3398]: 2013/06/03_13:20:30 debug: hb_send_local_status() {
heartbeat[3398]: 2013/06/03_13:20:30 debug: PID 3398: Sending local status curnode = 807aaec status: active
heartbeat[3398]: 2013/06/03_13:20:30 debug: process_clustermsg: node [SEVER2.domain]
heartbeat[3398]: 2013/06/03_13:20:30 debug: }/*hb_send_local_status*/;
heartbeat[3422]: 2013/06/03_13:20:30 debug: Packet authenticated
heartbeat[3423]: 2013/06/03_13:20:30 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:20:30 debug: read_child_dispatch() {
heartbeat[3398]: 2013/06/03_13:20:30 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:20:30 debug: process_clustermsg: node [192.168.0.1]
heartbeat[3398]: 2013/06/03_13:20:30 debug: }/*read_child_dispatch*/;
heartbeat[3398]: 2013/06/03_13:20:36 debug: read_child_dispatch() {
heartbeat[3398]: 2013/06/03_13:20:36 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:20:36 debug: process_clustermsg: node [SEVER1.domain]
heartbeat[3398]: 2013/06/03_13:20:36 debug: }/*read_child_dispatch*/;
heartbeat[3398]: 2013/06/03_13:20:40 debug: hb_send_local_status() {
heartbeat[3398]: 2013/06/03_13:20:40 debug: PID 3398: Sending local status curnode = 807aaec status: active
heartbeat[3398]: 2013/06/03_13:20:40 debug: process_clustermsg: node [SEVER2.domain]
heartbeat[3398]: 2013/06/03_13:20:40 debug: }/*hb_send_local_status*/;
heartbeat[3422]: 2013/06/03_13:20:40 debug: Packet authenticated
heartbeat[3423]: 2013/06/03_13:20:40 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:20:40 debug: read_child_dispatch() {
heartbeat[3398]: 2013/06/03_13:20:40 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:20:40 debug: process_clustermsg: node [192.168.0.1]
heartbeat[3398]: 2013/06/03_13:20:40 debug: }/*read_child_dispatch*/;
heartbeat[3398]: 2013/06/03_13:20:50 debug: hb_send_local_status() {
heartbeat[3398]: 2013/06/03_13:20:50 debug: PID 3398: Sending local status curnode = 807aaec status: active
heartbeat[3398]: 2013/06/03_13:20:50 debug: process_clustermsg: node [SEVER2.domain]
heartbeat[3398]: 2013/06/03_13:20:50 debug: }/*hb_send_local_status*/;
heartbeat[3422]: 2013/06/03_13:20:50 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:20:50 WARN: Gmain_timeout_dispatch: Dispatch function for send local status took too long to execute: 100 ms (> 50 ms) (GSource: 0x9924788)
heartbeat[3423]: 2013/06/03_13:20:50 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:20:50 debug: read_child_dispatch() {
heartbeat[3398]: 2013/06/03_13:20:50 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:20:50 debug: process_clustermsg: node [192.168.0.1]
heartbeat[3398]: 2013/06/03_13:20:50 debug: }/*read_child_dispatch*/;
heartbeat[3398]: 2013/06/03_13:21:00 debug: hb_send_local_status() {
heartbeat[3398]: 2013/06/03_13:21:00 debug: PID 3398: Sending local status curnode = 807aaec status: active
heartbeat[3398]: 2013/06/03_13:21:00 debug: process_clustermsg: node [SEVER2.domain]
heartbeat[3398]: 2013/06/03_13:21:00 debug: }/*hb_send_local_status*/;
heartbeat[3422]: 2013/06/03_13:21:00 debug: Packet authenticated
heartbeat[3423]: 2013/06/03_13:21:00 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:21:00 debug: read_child_dispatch() {
heartbeat[3398]: 2013/06/03_13:21:00 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:21:00 debug: process_clustermsg: node [192.168.0.1]
heartbeat[3398]: 2013/06/03_13:21:00 debug: }/*read_child_dispatch*/;
heartbeat[3398]: 2013/06/03_13:21:10 debug: hb_send_local_status() {
heartbeat[3398]: 2013/06/03_13:21:10 debug: PID 3398: Sending local status curnode = 807aaec status: active
heartbeat[3398]: 2013/06/03_13:21:10 debug: process_clustermsg: node [SEVER2.domain]
heartbeat[3398]: 2013/06/03_13:21:10 debug: }/*hb_send_local_status*/;
heartbeat[3422]: 2013/06/03_13:21:10 debug: Packet authenticated
heartbeat[3423]: 2013/06/03_13:21:10 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:21:10 debug: read_child_dispatch() {
heartbeat[3398]: 2013/06/03_13:21:10 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:21:10 debug: process_clustermsg: node [192.168.0.1]
heartbeat[3398]: 2013/06/03_13:21:10 debug: }/*read_child_dispatch*/;
heartbeat[3398]: 2013/06/03_13:21:20 debug: hb_send_local_status() {
heartbeat[3398]: 2013/06/03_13:21:20 debug: PID 3398: Sending local status curnode = 807aaec status: active
heartbeat[3398]: 2013/06/03_13:21:20 debug: process_clustermsg: node [SEVER2.domain]
heartbeat[3398]: 2013/06/03_13:21:20 debug: process_clustermsg: node [SEVER2.domain]
heartbeat[3398]: 2013/06/03_13:21:20 debug: }/*hb_send_local_status*/;
heartbeat[3422]: 2013/06/03_13:21:20 debug: Packet authenticated
heartbeat[3422]: 2013/06/03_13:21:21 debug: Packet authenticated
heartbeat[3423]: 2013/06/03_13:21:21 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:21:21 debug: read_child_dispatch() {
heartbeat[3398]: 2013/06/03_13:21:21 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:21:21 debug: process_clustermsg: node [192.168.0.1]
heartbeat[3398]: 2013/06/03_13:21:21 debug: }/*read_child_dispatch*/;
heartbeat[3398]: 2013/06/03_13:21:30 debug: hb_send_local_status() {
heartbeat[3398]: 2013/06/03_13:21:30 debug: PID 3398: Sending local status curnode = 807aaec status: active
heartbeat[3398]: 2013/06/03_13:21:30 debug: process_clustermsg: node [SEVER2.domain]
heartbeat[3398]: 2013/06/03_13:21:30 debug: }/*hb_send_local_status*/;
heartbeat[3422]: 2013/06/03_13:21:30 debug: Packet authenticated
heartbeat[3423]: 2013/06/03_13:21:30 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:21:30 debug: read_child_dispatch() {
heartbeat[3398]: 2013/06/03_13:21:31 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:21:31 debug: process_clustermsg: node [192.168.0.1]
heartbeat[3398]: 2013/06/03_13:21:31 debug: }/*read_child_dispatch*/;
heartbeat[3398]: 2013/06/03_13:21:37 WARN: node SEVER1.domain: is dead
ipfail[3604]: 2013/06/03_13:21:37 info: Status update: Node SEVER1.domain now has status dead
heartbeat[3398]: 2013/06/03_13:21:37 debug: process_clustermsg: node [SEVER2.domain]
heartbeat[3398]: 2013/06/03_13:21:37 debug: Queueing remote resource request (hook = 0x0x99139a0) stonith
heartbeat[3398]: 2013/06/03_13:21:37 debug: MSG: Dumping message with 11 fields
heartbeat[3398]: 2013/06/03_13:21:37 debug: MSG[0] : [t=stonith]
heartbeat[3398]: 2013/06/03_13:21:37 debug: MSG[1] : [node=SEVER1.domain]
heartbeat[3398]: 2013/06/03_13:21:37 debug: MSG[2] : [result=n_stnth]
heartbeat[3398]: 2013/06/03_13:21:37 debug: MSG[3] : [src=SEVER2.domain]
heartbeat[3398]: 2013/06/03_13:21:37 debug: MSG[4] : [(1)srcuuid=0x9924538(36 27)]
heartbeat[3398]: 2013/06/03_13:21:37 debug: MSG[5] : [seq=17]
heartbeat[3398]: 2013/06/03_13:21:37 debug: MSG[6] : [hg=519e1853]
heartbeat[3398]: 2013/06/03_13:21:37 debug: MSG[7] : [ts=51ac19d1]
heartbeat[3398]: 2013/06/03_13:21:37 debug: MSG[8] : [ld=1.40 0.69 0.27 2/220 4322]
heartbeat[3398]: 2013/06/03_13:21:37 debug: MSG[9] : [ttl=4]
heartbeat[3398]: 2013/06/03_13:21:37 debug: MSG[10] : [auth=1 6f4742b1]
heartbeat[3398]: 2013/06/03_13:21:37 debug: FilterNotifications(stonith) => 0
heartbeat[3398]: 2013/06/03_13:21:37 debug: stonith: child process unneeded.
heartbeat[3398]: 2013/06/03_13:21:37 debug: MSG: Dumping message with 11 fields
heartbeat[3398]: 2013/06/03_13:21:37 debug: MSG[0] : [t=stonith]
heartbeat[3398]: 2013/06/03_13:21:37 debug: MSG[1] : [node=SEVER1.domain]
heartbeat[3398]: 2013/06/03_13:21:37 debug: MSG[2] : [result=n_stnth]
heartbeat[3398]: 2013/06/03_13:21:37 debug: MSG[3] : [src=SEVER2.domain]
heartbeat[3398]: 2013/06/03_13:21:37 debug: MSG[4] : [(1)srcuuid=0x9924538(36 27)]
heartbeat[3398]: 2013/06/03_13:21:37 debug: MSG[5] : [seq=17]
heartbeat[3398]: 2013/06/03_13:21:37 debug: MSG[6] : [hg=519e1853]
heartbeat[3398]: 2013/06/03_13:21:37 debug: MSG[7] : [ts=51ac19d1]
heartbeat[3398]: 2013/06/03_13:21:37 debug: MSG[8] : [ld=1.40 0.69 0.27 2/220 4322]
heartbeat[3398]: 2013/06/03_13:21:37 debug: MSG[9] : [ttl=4]
heartbeat[3398]: 2013/06/03_13:21:37 debug: MSG[10] : [auth=1 6f4742b1]
heartbeat[3398]: 2013/06/03_13:21:37 WARN: No STONITH device configured.
heartbeat[3422]: 2013/06/03_13:21:37 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:21:37 WARN: Shared disks are not protected.
heartbeat[3398]: 2013/06/03_13:21:37 info: Resources being acquired from SEVER1.domain.
heartbeat[3398]: 2013/06/03_13:21:37 debug: Queueing remote resource request (hook = 0x0x99139c0) status
heartbeat[3398]: 2013/06/03_13:21:37 debug: MSG: Dumping message with 5 fields
heartbeat[3398]: 2013/06/03_13:21:37 debug: MSG[0] : [t=status]
heartbeat[3398]: 2013/06/03_13:21:37 debug: MSG[1] : [seq=1]
heartbeat[3398]: 2013/06/03_13:21:37 debug: MSG[2] : [ts=51ac19d1]
heartbeat[3398]: 2013/06/03_13:21:37 debug: MSG[3] : [src=SEVER1.domain]
heartbeat[3398]: 2013/06/03_13:21:37 debug: MSG[4] : [st=dead]
heartbeat[3398]: 2013/06/03_13:21:37 debug: FilterNotifications(status) => 1
heartbeat[3398]: 2013/06/03_13:21:37 debug: StartNextRemoteRscReq() - calling hook
heartbeat[3398]: 2013/06/03_13:21:37 debug: PerformQueuedNotifyWorld() msg follows
heartbeat[3398]: 2013/06/03_13:21:37 debug: MSG: Dumping message with 5 fields
heartbeat[3398]: 2013/06/03_13:21:37 debug: MSG[0] : [t=status]
heartbeat[3398]: 2013/06/03_13:21:37 debug: MSG[1] : [seq=1]
heartbeat[3398]: 2013/06/03_13:21:37 debug: MSG[2] : [ts=51ac19d1]
heartbeat[3398]: 2013/06/03_13:21:37 debug: MSG[3] : [src=SEVER1.domain]
heartbeat[3398]: 2013/06/03_13:21:37 debug: MSG[4] : [st=dead]
heartbeat[3398]: 2013/06/03_13:21:37 debug: FilterNotifications(status) => 1
heartbeat[3398]: 2013/06/03_13:21:37 debug: notify_world: invoking harc: OLD status: active
heartbeat[3398]: 2013/06/03_13:21:37 debug: Process [status] started pid 4323
heartbeat[4323]: 2013/06/03_13:21:37 debug: notify_world: setting SIGCHLD Handler to SIG_DFL
heartbeat[3398]: 2013/06/03_13:21:37 debug: Starting notify process [status]
heartbeat[4323]: 2013/06/03_13:21:37 debug: notify_world: Running harc status
heartbeat[3398]: 2013/06/03_13:21:37 debug: takeover_from_node: other now stable
heartbeat[3398]: 2013/06/03_13:21:37 debug: Process [req_our_resources] started pid 4324
heartbeat[4324]: 2013/06/03_13:21:37 debug: req_our_resources(/usr/share/heartbeat/ResourceManager listkeys SEVER2.domain)
heartbeat[3398]: 2013/06/03_13:21:37 info: Link SEVER1.domain:eth1 dead.
heartbeat[4324]: 2013/06/03_13:21:37 debug: req_our_resources() before fgets()
heartbeat[3398]: 2013/06/03_13:21:37 debug: Queueing remote resource request (hook = 0x0x9913a00) ifstat
heartbeat[3398]: 2013/06/03_13:21:37 debug: MSG: Dumping message with 4 fields
heartbeat[3398]: 2013/06/03_13:21:37 debug: MSG[0] : [t=ifstat]
heartbeat[4324]: 2013/06/03_13:21:37 debug: req_our_resources() fgets => NULL
heartbeat[3398]: 2013/06/03_13:21:37 debug: MSG[1] : [node=SEVER1.domain]
heartbeat[4324]: 2013/06/03_13:21:37 info: No local resources [/usr/share/heartbeat/ResourceManager listkeys SEVER2.domain] to acquire.
heartbeat[3398]: 2013/06/03_13:21:37 debug: MSG[2] : [ifname=eth1]
heartbeat[3398]: 2013/06/03_13:21:37 debug: MSG[3] : [st=dead]
heartbeat[4324]: 2013/06/03_13:21:37 debug: Sending hold resources msg: all, stable=1 # req_our_resources()
heartbeat[3398]: 2013/06/03_13:21:37 debug: FilterNotifications(ifstat) => 0
heartbeat[4324]: 2013/06/03_13:21:37 info: Writing type [resource] message to FIFO
heartbeat[3398]: 2013/06/03_13:21:37 debug: ifstat: child process unneeded.
heartbeat[3398]: 2013/06/03_13:21:37 debug: MSG: Dumping message with 4 fields
heartbeat[3419]: 2013/06/03_13:21:37 debug: fifo_child message:
heartbeat[4324]: 2013/06/03_13:21:37 info: FIFO message [type resource] written rc=79
heartbeat[3398]: 2013/06/03_13:21:37 debug: MSG[0] : [t=ifstat]
heartbeat[3419]: 2013/06/03_13:21:37 debug: MSG: Dumping message with 5 fields
heartbeat[3398]: 2013/06/03_13:21:37 debug: MSG[1] : [node=SEVER1.domain]
heartbeat[3419]: 2013/06/03_13:21:37 debug: MSG[0] : [t=resource]
heartbeat[3398]: 2013/06/03_13:21:37 debug: MSG[2] : [ifname=eth1]
heartbeat[3419]: 2013/06/03_13:21:37 debug: MSG[1] : [rsc_hold=all]
harc[4323]: 2013/06/03_13:21:37 info: Running /etc/ha.d/rc.d/status status
heartbeat[3398]: 2013/06/03_13:21:37 debug: MSG[3] : [st=dead]
heartbeat[3419]: 2013/06/03_13:21:37 debug: MSG[2] : [rtype=full]
heartbeat[3398]: 2013/06/03_13:21:37 info: Managed req_our_resources process 4324 exited with return code 0.
heartbeat[3419]: 2013/06/03_13:21:37 debug: MSG[3] : [isstable=1]
heartbeat[3419]: 2013/06/03_13:21:37 debug: MSG[4] : [info=req_our_resources()]
heartbeat[3398]: 2013/06/03_13:21:37 debug: RscMgmtProc 'req_our_resources' exited code 0
heartbeat[3398]: 2013/06/03_13:21:37 info: AnnounceTakeover(local 1, foreign 1, reason 'req_our_resources' (1))
heartbeat[3398]: 2013/06/03_13:21:37 debug: StartNextRemoteRscReq(): child count 1
heartbeat[3398]: 2013/06/03_13:21:37 debug: FIFO_child_msg_dispatch() {
heartbeat[3398]: 2013/06/03_13:21:37 debug: process_clustermsg: node [SEVER2.domain]
heartbeat[3398]: 2013/06/03_13:21:37 info: AnnounceTakeover(local 1, foreign 1, reason 'T_RESOURCES(us)' (1))
heartbeat[3398]: 2013/06/03_13:21:37 debug: hb_rsc_isstable: ResourceMgmt_child_count: 1, other_is_stable: 1, takeover_in_progress: 1, going_standby: 0, standby running(ms): 0, resourcestate: 4
heartbeat[3398]: 2013/06/03_13:21:37 debug: }/*FIFO_child_msg_dispatch*/;
heartbeat[3422]: 2013/06/03_13:21:37 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:21:37 debug: APIclients_input_dispatch() {
heartbeat[3398]: 2013/06/03_13:21:37 debug: ProcessAnAPIRequest() {
heartbeat[3398]: 2013/06/03_13:21:37 debug:  return TRUE;
heartbeat[3398]: 2013/06/03_13:21:37 debug: }/*ProcessAnAPIRequest*/;
heartbeat[3398]: 2013/06/03_13:21:37 debug: return 1;
heartbeat[3398]: 2013/06/03_13:21:37 debug: }/*APIclients_input_dispatch*/;
heartbeat[3398]: 2013/06/03_13:21:37 debug: APIclients_input_dispatch() {
heartbeat[3398]: 2013/06/03_13:21:37 debug: ProcessAnAPIRequest() {
heartbeat[3398]: 2013/06/03_13:21:37 debug:  return TRUE;
ipfail[3604]: 2013/06/03_13:21:37 debug: Found ping node 192.168.0.1!
heartbeat[3398]: 2013/06/03_13:21:37 debug: }/*ProcessAnAPIRequest*/;
heartbeat[3398]: 2013/06/03_13:21:37 debug: return 1;
heartbeat[3398]: 2013/06/03_13:21:37 debug: }/*APIclients_input_dispatch*/;
heartbeat[3398]: 2013/06/03_13:21:37 debug: APIclients_input_dispatch() {
heartbeat[3398]: 2013/06/03_13:21:37 debug: ProcessAnAPIRequest() {
heartbeat[3398]: 2013/06/03_13:21:37 debug:  return TRUE;
heartbeat[3398]: 2013/06/03_13:21:37 debug: }/*ProcessAnAPIRequest*/;
heartbeat[3398]: 2013/06/03_13:21:37 debug: return 1;
heartbeat[3398]: 2013/06/03_13:21:37 debug: }/*APIclients_input_dispatch*/;
heartbeat[3398]: 2013/06/03_13:21:37 debug: APIclients_input_dispatch() {
heartbeat[3398]: 2013/06/03_13:21:37 debug: ProcessAnAPIRequest() {
heartbeat[3398]: 2013/06/03_13:21:37 debug:  return TRUE;
ipfail[3604]: 2013/06/03_13:21:37 info: NS: We are still alive!
mach_down[4352]: 2013/06/03_13:21:37 info: Taking over resource group drbddisk
heartbeat[3398]: 2013/06/03_13:21:37 debug: }/*ProcessAnAPIRequest*/;
heartbeat[3398]: 2013/06/03_13:21:37 debug: return 1;
heartbeat[3398]: 2013/06/03_13:21:37 debug: }/*APIclients_input_dispatch*/;
ipfail[3604]: 2013/06/03_13:21:37 info: Link Status update: Link SEVER1.domain/eth1 now has status dead
heartbeat[3398]: 2013/06/03_13:21:37 debug: APIclients_input_dispatch() {
heartbeat[3398]: 2013/06/03_13:21:37 debug: ProcessAnAPIRequest() {
ResourceManager[4378]: 2013/06/03_13:21:37 info: Acquiring resource group: SEVER1.domain drbddisk Filesystem::/dev/drbd0::/usr1::ext3 httpd postgresql 192.168.0.110/24 MailTo::test_shuog****@yahoo*****::SEVER_FailOver
heartbeat[3398]: 2013/06/03_13:21:37 debug:  return TRUE;
heartbeat[3398]: 2013/06/03_13:21:37 debug: }/*ProcessAnAPIRequest*/;
heartbeat[3398]: 2013/06/03_13:21:37 debug: return 1;
heartbeat[3398]: 2013/06/03_13:21:37 debug: }/*APIclients_input_dispatch*/;
heartbeat[3398]: 2013/06/03_13:21:37 debug: APIclients_input_dispatch() {
heartbeat[3398]: 2013/06/03_13:21:37 debug: ProcessAnAPIRequest() {
heartbeat[3398]: 2013/06/03_13:21:37 debug:  return TRUE;
ipfail[3604]: 2013/06/03_13:21:37 debug: Found ping node 192.168.0.1!
heartbeat[3398]: 2013/06/03_13:21:37 debug: }/*ProcessAnAPIRequest*/;
heartbeat[3398]: 2013/06/03_13:21:37 debug: return 1;
heartbeat[3398]: 2013/06/03_13:21:37 debug: }/*APIclients_input_dispatch*/;
heartbeat[3398]: 2013/06/03_13:21:37 debug: APIclients_input_dispatch() {
heartbeat[3398]: 2013/06/03_13:21:37 debug: ProcessAnAPIRequest() {
heartbeat[3398]: 2013/06/03_13:21:37 debug:  return TRUE;
heartbeat[3398]: 2013/06/03_13:21:37 debug: }/*ProcessAnAPIRequest*/;
heartbeat[3398]: 2013/06/03_13:21:37 debug: return 1;
heartbeat[3398]: 2013/06/03_13:21:37 debug: }/*APIclients_input_dispatch*/;
heartbeat[3398]: 2013/06/03_13:21:37 debug: APIclients_input_dispatch() {
heartbeat[3398]: 2013/06/03_13:21:37 debug: ProcessAnAPIRequest() {
heartbeat[3398]: 2013/06/03_13:21:37 debug:  return TRUE;
ipfail[3604]: 2013/06/03_13:21:37 info: Asking other side for ping node count.
heartbeat[3398]: 2013/06/03_13:21:37 debug: }/*ProcessAnAPIRequest*/;
ResourceManager[4378]: 2013/06/03_13:21:37 info: Running /etc/ha.d/resource.d/drbddisk  start
heartbeat[3398]: 2013/06/03_13:21:37 debug: return 1;
ipfail[3604]: 2013/06/03_13:21:37 debug: Message [num_ping] sent.
heartbeat[3398]: 2013/06/03_13:21:37 debug: }/*APIclients_input_dispatch*/;
ipfail[3604]: 2013/06/03_13:21:37 info: Checking remote count of ping nodes.
heartbeat[3398]: 2013/06/03_13:21:37 debug: APIclients_input_dispatch() {
heartbeat[3398]: 2013/06/03_13:21:37 debug: ProcessAnAPIRequest() {
heartbeat[3398]: 2013/06/03_13:21:37 debug: Sending API message to cluster...
heartbeat[3398]: 2013/06/03_13:21:37 debug: MSG: Dumping message with 6 fields
ResourceManager[4378]: 2013/06/03_13:21:37 debug: Starting /etc/ha.d/resource.d/drbddisk  start
heartbeat[3398]: 2013/06/03_13:21:37 debug: MSG[0] : [t=num_ping_nodes]
heartbeat[3398]: 2013/06/03_13:21:37 debug: MSG[1] : [src=SEVER2.domain]
heartbeat[3398]: 2013/06/03_13:21:37 debug: MSG[2] : [num_ping=1]
heartbeat[3398]: 2013/06/03_13:21:37 debug: MSG[3] : [dest=SEVER1.domain]
heartbeat[3398]: 2013/06/03_13:21:37 debug: MSG[4] : [from_id=ipfail]
heartbeat[3398]: 2013/06/03_13:21:37 debug: MSG[5] : [to_id=ipfail]
heartbeat[3398]: 2013/06/03_13:21:37 debug: process_clustermsg: node [SEVER2.domain]
heartbeat[3398]: 2013/06/03_13:21:37 debug:  return TRUE;
heartbeat[3422]: 2013/06/03_13:21:37 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:21:37 debug: }/*ProcessAnAPIRequest*/;
heartbeat[3398]: 2013/06/03_13:21:37 debug: return 1;
heartbeat[3398]: 2013/06/03_13:21:37 debug: }/*APIclients_input_dispatch*/;
State change failed: (-1) Multiple primaries not allowed by config
Command '/sbin/drbdsetup /dev/drbd0 primary' terminated with exit code 11
drbdsetup exited with code 11
heartbeat[3398]: 2013/06/03_13:21:40 debug: hb_send_local_status() {
heartbeat[3398]: 2013/06/03_13:21:40 debug: PID 3398: Sending local status curnode = 807aaec status: active
heartbeat[3398]: 2013/06/03_13:21:40 debug: process_clustermsg: node [SEVER2.domain]
heartbeat[3398]: 2013/06/03_13:21:40 debug: }/*hb_send_local_status*/;
heartbeat[3422]: 2013/06/03_13:21:40 debug: Packet authenticated
heartbeat[3423]: 2013/06/03_13:21:40 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:21:41 debug: read_child_dispatch() {
heartbeat[3398]: 2013/06/03_13:21:41 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:21:41 debug: process_clustermsg: node [192.168.0.1]
heartbeat[3398]: 2013/06/03_13:21:41 debug: }/*read_child_dispatch*/;
State change failed: (-1) Multiple primaries not allowed by config
Command '/sbin/drbdsetup /dev/drbd0 primary' terminated with exit code 11
drbdsetup exited with code 11
State change failed: (-1) Multiple primaries not allowed by config
Command '/sbin/drbdsetup /dev/drbd0 primary' terminated with exit code 11
drbdsetup exited with code 11
IPaddr[4427]: 2013/06/03_13:21:44 INFO:  Resource is stopped
State change failed: (-1) Multiple primaries not allowed by config
Command '/sbin/drbdsetup /dev/drbd0 primary' terminated with exit code 11
drbdsetup exited with code 11
State change failed: (-1) Multiple primaries not allowed by config
Command '/sbin/drbdsetup /dev/drbd0 primary' terminated with exit code 11
drbdsetup exited with code 11
State change failed: (-1) Multiple primaries not allowed by config
Command '/sbin/drbdsetup /dev/drbd0 primary' terminated with exit code 11
drbdsetup exited with code 11
ResourceManager[4378]: 2013/06/03_13:21:50 debug: /etc/ha.d/resource.d/drbddisk  start done. RC=20
ResourceManager[4378]: 2013/06/03_13:21:50 ERROR: Return code 20 from /etc/ha.d/resource.d/drbddisk
ResourceManager[4378]: 2013/06/03_13:21:50 CRIT: Giving up resources due to failure of drbddisk
ResourceManager[4378]: 2013/06/03_13:21:50 info: Releasing resource group: SEVER1.domain drbddisk Filesystem::/dev/drbd0::/usr1::ext3 httpd postgresql 192.168.0.110/24 MailTo::test_shuog****@yahoo*****::SEVER_FailOver
ResourceManager[4378]: 2013/06/03_13:21:50 info: Running /etc/ha.d/resource.d/MailTo test****@yahoo***** SEVER_FailOver stop
ResourceManager[4378]: 2013/06/03_13:21:50 debug: Starting /etc/ha.d/resource.d/MailTo test****@yahoo***** SEVER_FailOver stop
heartbeat[3398]: 2013/06/03_13:21:50 debug: hb_send_local_status() {
heartbeat[3398]: 2013/06/03_13:21:50 debug: PID 3398: Sending local status curnode = 807aaec status: active
heartbeat[3398]: 2013/06/03_13:21:50 debug: process_clustermsg: node [SEVER2.domain]
heartbeat[3398]: 2013/06/03_13:21:50 debug: }/*hb_send_local_status*/;
heartbeat[3422]: 2013/06/03_13:21:50 debug: Packet authenticated
heartbeat[3423]: 2013/06/03_13:21:50 debug: Packet authenticated
MailTo[4514]: 2013/06/03_13:21:50 INFO:  Success
INFO:  Success
ResourceManager[4378]: 2013/06/03_13:21:50 debug: /etc/ha.d/resource.d/MailTo test****@yahoo***** SEVER_FailOver stop done. RC=0
heartbeat[3398]: 2013/06/03_13:21:51 debug: read_child_dispatch() {
heartbeat[3398]: 2013/06/03_13:21:51 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:21:51 debug: process_clustermsg: node [192.168.0.1]
heartbeat[3398]: 2013/06/03_13:21:51 debug: }/*read_child_dispatch*/;
ResourceManager[4378]: 2013/06/03_13:21:51 info: Running /etc/ha.d/resource.d/IPaddr 192.168.0.110/24 stop
ResourceManager[4378]: 2013/06/03_13:21:51 debug: Starting /etc/ha.d/resource.d/IPaddr 192.168.0.110/24 stop
In IP Stop
IPaddr[4570]: 2013/06/03_13:21:51 INFO:  Success
INFO:  Success
ResourceManager[4378]: 2013/06/03_13:21:51 debug: /etc/ha.d/resource.d/IPaddr 192.168.0.110/24 stop done. RC=0
ResourceManager[4378]: 2013/06/03_13:21:51 info: Running /etc/init.d/postgresql  stop
ResourceManager[4378]: 2013/06/03_13:21:51 debug: Starting /etc/init.d/postgresql  stop
Stopping postgresql service: [FAILED]
ResourceManager[4378]: 2013/06/03_13:21:52 debug: /etc/init.d/postgresql  stop done. RC=1
ResourceManager[4378]: 2013/06/03_13:21:52 ERROR: Return code 1 from /etc/init.d/postgresql
ResourceManager[4378]: 2013/06/03_13:21:53 info: Retrying failed stop operation [postgresql]
ResourceManager[4378]: 2013/06/03_13:21:53 info: Running /etc/init.d/postgresql  stop
ResourceManager[4378]: 2013/06/03_13:21:53 debug: Starting /etc/init.d/postgresql  stop
Stopping postgresql service: [FAILED]
ResourceManager[4378]: 2013/06/03_13:21:53 debug: /etc/init.d/postgresql  stop done. RC=1
ResourceManager[4378]: 2013/06/03_13:21:53 ERROR: Return code 1 from /etc/init.d/postgresql
ResourceManager[4378]: 2013/06/03_13:21:54 info: Retrying failed stop operation [postgresql]
ResourceManager[4378]: 2013/06/03_13:21:54 info: Running /etc/init.d/postgresql  stop
ResourceManager[4378]: 2013/06/03_13:21:54 debug: Starting /etc/init.d/postgresql  stop
Stopping postgresql service: [FAILED]
ResourceManager[4378]: 2013/06/03_13:21:54 debug: /etc/init.d/postgresql  stop done. RC=1
ResourceManager[4378]: 2013/06/03_13:21:54 ERROR: Return code 1 from /etc/init.d/postgresql
ResourceManager[4378]: 2013/06/03_13:21:56 info: Retrying failed stop operation [postgresql]
ResourceManager[4378]: 2013/06/03_13:21:56 info: Running /etc/init.d/postgresql  stop
ResourceManager[4378]: 2013/06/03_13:21:56 debug: Starting /etc/init.d/postgresql  stop
Stopping postgresql service: [FAILED]
ResourceManager[4378]: 2013/06/03_13:21:56 debug: /etc/init.d/postgresql  stop done. RC=1
ResourceManager[4378]: 2013/06/03_13:21:56 ERROR: Return code 1 from /etc/init.d/postgresql
ResourceManager[4378]: 2013/06/03_13:21:57 info: Retrying failed stop operation [postgresql]
ResourceManager[4378]: 2013/06/03_13:21:57 info: Running /etc/init.d/postgresql  stop
ResourceManager[4378]: 2013/06/03_13:21:57 debug: Starting /etc/init.d/postgresql  stop
Stopping postgresql service: [FAILED]
ResourceManager[4378]: 2013/06/03_13:21:57 debug: /etc/init.d/postgresql  stop done. RC=1
ResourceManager[4378]: 2013/06/03_13:21:57 ERROR: Return code 1 from /etc/init.d/postgresql
ResourceManager[4378]: 2013/06/03_13:21:58 info: Retrying failed stop operation [postgresql]
ResourceManager[4378]: 2013/06/03_13:21:59 info: Running /etc/init.d/postgresql  stop
ResourceManager[4378]: 2013/06/03_13:21:59 debug: Starting /etc/init.d/postgresql  stop
Stopping postgresql service: [FAILED]
ResourceManager[4378]: 2013/06/03_13:21:59 debug: /etc/init.d/postgresql  stop done. RC=1
ResourceManager[4378]: 2013/06/03_13:21:59 ERROR: Return code 1 from /etc/init.d/postgresql
ResourceManager[4378]: 2013/06/03_13:22:00 info: Retrying failed stop operation [postgresql]
ResourceManager[4378]: 2013/06/03_13:22:00 info: Running /etc/init.d/postgresql  stop
ResourceManager[4378]: 2013/06/03_13:22:00 debug: Starting /etc/init.d/postgresql  stop
Stopping postgresql service: [FAILED]
ResourceManager[4378]: 2013/06/03_13:22:00 debug: /etc/init.d/postgresql  stop done. RC=1
ResourceManager[4378]: 2013/06/03_13:22:00 ERROR: Return code 1 from /etc/init.d/postgresql
heartbeat[3398]: 2013/06/03_13:22:00 debug: hb_send_local_status() {
heartbeat[3398]: 2013/06/03_13:22:00 debug: PID 3398: Sending local status curnode = 807aaec status: active
heartbeat[3398]: 2013/06/03_13:22:00 debug: process_clustermsg: node [SEVER2.domain]
heartbeat[3398]: 2013/06/03_13:22:00 debug: }/*hb_send_local_status*/;
heartbeat[3422]: 2013/06/03_13:22:00 debug: Packet authenticated
heartbeat[3423]: 2013/06/03_13:22:00 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:22:01 debug: read_child_dispatch() {
heartbeat[3398]: 2013/06/03_13:22:01 debug: Packet authenticated
heartbeat[3398]: 2013/06/03_13:22:01 debug: process_clustermsg: node [192.168.0.1]
heartbeat[3398]: 2013/06/03_13:22:01 debug: }/*read_child_dispatch*/;
ResourceManager[4378]: 2013/06/03_13:22:01 info: Retrying failed stop operation [postgresql]
ResourceManager[4378]: 2013/06/03_13:22:01 info: Running /etc/init.d/postgresql  stop
ResourceManager[4378]: 2013/06/03_13:22:01 debug: Starting /etc/init.d/postgresql  stop
Stopping postgresql service: [FAILED]
ResourceManager[4378]: 2013/06/03_13:22:02 debug: /etc/init.d/postgresql  stop done. RC=1
ResourceManager[4378]: 2013/06/03_13:22:02 ERROR: Return code 1 from /etc/init.d/postgresql
ResourceManager[4378]: 2013/06/03_13:22:03 info: Retrying failed stop operation [postgresql]
ResourceManager[4378]: 2013/06/03_13:22:03 info: Running /etc/init.d/postgresql  stop
ResourceManager[4378]: 2013/06/03_13:22:03 debug: Starting /etc/init.d/postgresql  stop
Stopping postgresql service: [FAILED]
ResourceManager[4378]: 2013/06/03_13:22:03 debug: /etc/init.d/postgresql  stop done. RC=1
ResourceManager[4378]: 2013/06/03_13:22:03 ERROR: Return code 1 from /etc/init.d/postgresql
ResourceManager[4378]: 2013/06/03_13:22:04 info: Retrying failed stop operation [postgresql]
ResourceManager[4378]: 2013/06/03_13:22:04 info: Running /etc/init.d/postgresql  stop
ResourceManager[4378]: 2013/06/03_13:22:04 debug: Starting /etc/init.d/postgresql  stop
Stopping postgresql service: [FAILED]
ResourceManager[4378]: 2013/06/03_13:22:04 debug: /etc/init.d/postgresql  stop done. RC=1
ResourceManager[4378]: 2013/06/03_13:22:05 ERROR: Return code 1 from /etc/init.d/postgresql
ResourceManager[4378]: 2013/06/03_13:22:06 info: Retrying failed stop operation [postgresql]
ResourceManager[4378]: 2013/06/03_13:22:06 info: Running /etc/init.d/postgresql  stop
ResourceManager[4378]: 2013/06/03_13:22:06 debug: Starting /etc/init.d/postgresql  stop
Stopping postgresql service: [FAILED]
ResourceManager[4378]: 2013/06/03_13:22:06 debug: /etc/init.d/postgresql  stop done. RC=1
ResourceManager[4378]: 2013/06/03_13:22:06 ERROR: Return code 1 from /etc/init.d/postgresql
ResourceManager[4378]: 2013/06/03_13:22:06 CRIT: Resource STOP failure. Reboot required!
ResourceManager[4378]: 2013/06/03_13:22:06 CRIT: Killing heartbeat ungracefully!
以上です。
なにとぞ、よろしくお願い申し上げます。
        
----- Original Message -----
> From: "momok****@mail*****" <momok****@mail*****>
> To: delta_syste****@yahoo*****; linux****@lists*****
> Cc: 
> Date: 2013/6/3, Mon 23:44
> Subject: Re: [Linux-ha-jp] 待機系がフェールオーバーし、再起動を繰り返す
> 
> 広瀬です
> 
> ha-debug側のログも取られているようですので、そちらの当該時間帯の
> ログも提示していただけますでしょうか？
> 
> あと、haresourcesの中身が提示されたものと、ログの中が違う気がします。
> 
>>  3.haresourcesの抜粋
>>  SEVER1 IPaddr::192.168.0.110/24
> 
> 　↑
> この部分
> 　↓
> 
>>  ResourceManager[4378]: 2013/06/03_13:21:37 info: Acquiring resource 
> group: SEVER1.domain drbddisk Filesystem::/dev/drbd0::/usr1::ext3 httpd 
> postgresql 192.168.0.110/24 MailTo::test****@yahoo*****::server_FailOver
> 
> 
> あと、DRBD領域にはDBのデータがある、という事で間違いないでしょうか？
> 
> 
> よろしくお願い致します。
>

Linux-HA Japan Forkpm_logconv-cspm_diskdpm_logconv-hbpm_extrasdocpm_crmgenvm-ctlpm_kvm_tools

[Linux-ha-jp] 待機系がフェールオーバーし、再起動を繰り返す

Linux-HA Japan
Fork
pm_logconv-cs
pm_diskd
pm_logconv-hb
pm_extras
doc
pm_crmgen
vm-ctl
pm_kvm_tools