Jgroups Re/Joining issue if restarted at a later time

Help
2013-05-22
2013-06-02
  • Rajan Krishnan

    Rajan Krishnan - 2013-05-22

    Hello All
    We are using Jgroups 3.2.7. Here Node means an application with
    Jgroups in it. I have Node1, Node2 on physcial Machine 1 and Node3 and
    Nod4 on Physcial Machine 2.

    when i start all the nodes together all works great, cluster is formed
    with Node1, Node2, Node3 and Node4 as its members.

    Scenario 1: If I restart one of the nodes let us say Node1 when
    cluster is formed with Node1, Node2, Node3 and Node4. My cluster looks
    like
    Node1 sees only Node1, Node2, 5fbbafbf-769a-b553-d89a-eed703019334,
    ec3af265-f06f-5cff-ce3a-5391a01a0b19
    Node2 sees Node1, Node2, Node3 and Node4
    Node3 sess only Node2, Node3 and Node4
    Node4 sees only Node2, Node3 and Node4

    Scenario 2: If I restart two of the nodes let us say Node1 and Node2,
    when cluster is formed with Node1, Node2, Node3 and Node4. My cluster
    looks like
    Node1 sees only Node1, Node2
    Node2 sees Node1, Node2
    Node3 sess only Node3 and Node4
    Node4 sees only Node3 and Node4

    Scenario 3: If I start Node1, Node2, Node3 and leave behind Node4.
    After the cluster is formed with Node2, Node3 and Node4, I started
    Node1, now after Node1 is started here is what my cluster looks like

    Node1 sees only Node1, Node2, 5fbbafbf-769a-b553-d89a-eed703019334,
    ec3af265-f06f-5cff-ce3a-5391a01a0b19
    Node2 sees Node1, Node2, Node3 and Node4
    Node3 sees only Node2, Node3 and Node4
    Node4 sees only Node2, Node3 and Node4

    My Jgroups protocol config looks like

    UDP(mcast_addr=239.56.30.2;mcast_port=7500;ip_ttl=32;mcast_send_buf_size=64000;mcast_recv_buf_size=64000;ucast_send_buf_size=64000;ucast_recv_buf_size=64000;loopback=true;enable_diagnostics=false;discard_incompatible_packets=true)
    PING(timeout=2000;num_initial_members=3)
    MERGE2(min_interval=5000;max_interval=10000)
    FD_SOCK
    FD(timeout=5000;max_tries=5;shun=true)
    VERIFY_SUSPECT(timeout=30000)
    pbcast.NAKACK(gc_lag=50;retransmit_timeout=600,1200,2400,4800;log_not_found_msgs=false)
    UNICAST(timeout=600,1200,2400,4800)
    pbcast.STABLE(desired_avg_gossip=20000)
    FRAG(frag_size=32768)
    pbcast.GMS(join_timeout=5000;shun=false;print_local_addr=false)FC(max_credits=200000;down_thread=false)
    

    Tried changing frag_size as 8192, 4096 as well with no luck. We are
    using UDP IP multicast to form our clusters over Red hat linux.

    Anything odd in my protocol config, looks like rejoining is not
    happening as expected. Any help would be appreciated

    Note: if PING is disabled on the multicast will that create this issue
    Regards

     
    Last edit: Rajan Krishnan 2013-05-22
  • Bela Ban

    Bela Ban - 2013-05-22

    Your config is outdated (shun for instance was deprecated a long time ago).
    Try this again with the udp.xml config from 3.2.7.
    If it still fails, provide exact instructions on how to reproduce it.

     
  • Rajan Krishnan

    Rajan Krishnan - 2013-06-01

    Now my update config looks like the one below

    UDP(mcast_addr=239.56.30.2;mcast_port=7500;ip_ttl=32;mcast_send_buf_size=64000;mcast_recv_buf_size=64000;ucast_send_buf_size=64000;ucast_recv_buf_size=64000;loopback=true;enable_diagnostics=false;discard_incompatible_packets=true)
    PING(timeout=2000;num_initial_members=3)
    MERGE2(min_interval=5000;max_interval=10000)
    FD_SOCK
    FD(timeout=5000;max_tries=5)
    VERIFY_SUSPECT(timeout=30000)
    pbcast.NAKACK(retransmit_timeout=600,1200,2400,4800;log_not_found_msgs=false)
    UNICAST(timeout=600,1200,2400,4800)
    pbcast.STABLE(desired_avg_gossip=20000)
    FRAG(frag_size=8096)
    pbcast.GMS(join_timeout=5000;print_local_addr=false;max_join_attempts=5)FC(max_credits=200000;down_thread=false)

    And tried this below scenario

    Scenario 3:
    I have Node1, Node2 on physcial Machine 1 and Node3 and
    Node4 on Physcial Machine 2.

    If I start Node1, Node2, Node3 and leave behind Node4.
    After the cluster is formed with Node1, Node2 and Node3, I started
    Node4, now Node 4 is not able to join the cluster here is my Jgroups logs on node4,host name of node4 is leddvd0024 , ip address of host name is 54.56.30.28, multicast ip is 239.56.30.2

    Note: few warnings i need to fix in my UDP config as shown in the log

    Log from Node4:

    06/01/13 23:14:09 (0.00 ms)|[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'|AUDITING|NONE|TEST|NONE|Using jg-magic-map.xml as magic number file and jg-protocol-ids.xml for protocol IDs|
    06/01/13 23:14:09 (0.00 ms)|[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'|WARNING|NONE|TEST|NONE|[JGRP00013] TP.discard_incompatible_packets has been deprecated: incompatible packets are discarded anyway|
    06/01/13 23:14:09 (0.00 ms)|[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'|WARNING|NONE|TEST|NONE|[JGRP00013] UNICAST.setTimeout has been deprecated: not used anymore|
    06/01/13 23:14:09 (0.00 ms)|[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'|AUDITING|NONE|TEST|NONE|set property UDP.diagnostics_addr to default value /224.0.75.75|
    06/01/13 23:14:09 (0.00 ms)|[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'|AUDITING|NONE|TEST|NONE|received CONFIG event: {bind_addr=/54.56.30.28}|
    06/01/13 23:14:10 (0.00 ms)|[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'|VERBOSE|NONE|TEST|NONE|setting local_addr (null) to leddvd0024 -27944|
    06/01/13 23:14:10 (0.00 ms)|[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'|VERBOSE|NONE|TEST|NONE|stable task started|
    06/01/13 23:14:10 (0.00 ms)|[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'|AUDITING|NONE|TEST|NONE|sockets will use interface 54.56.30.28|
    06/01/13 23:14:10 (0.00 ms)|[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'|AUDITING|NONE|TEST|NONE|socket information:
    , mcast_addr=239.56.30.2:7500, bind_addr=/54.56.30.28, ttl=32
    sock: bound to 54.56.30.28:50535, receive buffer size=64000, send buffer size=64000
    mcast_sock: bound to 54.56.30.28:7500, send buffer size=64000, receive buffer size=64000|
    06/01/13 23:14:10 (0.00 ms)|[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'|AUDITING|NONE|TEST|NONE|address=leddvd0024 -27944, cluster=mockDev7, physical address=54.56.30.28:50535|
    06/01/13 23:14:10 (0.00 ms)|FD_SOCK acceptor,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|waiting for client connections on /54.56.30.28:48211|
    06/01/13 23:14:10 (0.00 ms)|[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'|VERBOSE|NONE|TEST|NONE|sending msg to null, src=leddvd0024 -27944, headers are PING: [PING: type=GET_MBRS_REQ, cluster=mockDev7, arg=leddvd0024 -27944, view_id=, is_server=false, is_coord=false, logical_name=leddvd0024 -27944, physical_addrs=54.56.30.28:50535], UDP: [channel_name=mockDev7]|
    06/01/13 23:14:10 (0.00 ms)|[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'|VERBOSE|NONE|TEST|NONE|looping back message [dst: <null>, src: leddvd0024 -27944 (2 headers), size=0 bytes, flags=OOB]|
    06/01/13 23:14:10 (0.00 ms)|OOB-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|received [dst: <null>, src: leddvd0024 -27944 (2 headers), size=0 bytes, flags=OOB], headers are PING: [PING: type=GET_MBRS_REQ, cluster=mockDev7, arg=leddvd0024 -27944, view_id=, is_server=false, is_coord=false, logical_name=leddvd0024 -27944, physical_addrs=54.56.30.28:50535], UDP: [channel_name=mockDev7]|
    06/01/13 23:14:10 (0.00 ms)|OOB-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|leddvd0024 -27944: received GET_MBRS_REQ from leddvd0024 -27944, sending response [PING: type=GET_MBRS_RSP, arg=leddvd0024 -27944, view_id=, is_server=false, is_coord=false, logical_name=leddvd0024 -27944, physical_addrs=54.56.30.28:50535]|
    06/01/13 23:14:10 (0.00 ms)|OOB-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|sending msg to leddvd0024 -27944, src=leddvd0024 -27944, headers are PING: [PING: type=GET_MBRS_RSP, arg=leddvd0024 -27944, view_id=, is_server=false, is_coord=false, logical_name=leddvd0024 -27944, physical_addrs=54.56.30.28:50535], UDP: [channel_name=mockDev7]|
    06/01/13 23:14:10 (0.00 ms)|OOB-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|looping back message [dst: leddvd0024 -27944, src: leddvd0024 -27944 (2 headers), size=0 bytes, flags=OOB]|
    06/01/13 23:14:10 (0.00 ms)|OOB-2,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|received [dst: leddvd0024 -27944, src: leddvd0024 -27944 (2 headers), size=0 bytes, flags=OOB], headers are PING: [PING: type=GET_MBRS_RSP, arg=leddvd0024 -27944, view_id=, is_server=false, is_coord=false, logical_name=leddvd0024 -27944, physical_addrs=54.56.30.28:50535], UDP: [channel_name=mockDev7]|
    06/01/13 23:14:10 (0.00 ms)|OOB-2,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|leddvd0024 -27944: received GET_MBRS_RSP from leddvd0024 -27944: leddvd0024 -27944, view_id=, is_server=false, is_coord=false, logical_name=leddvd0024 -27944, physical_addrs=54.56.30.28:50535|
    06/01/13 23:14:10 (0.00 ms)|OOB-2,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|received [dst: <null>, src: leddvd0024 -27944 (2 headers), size=0 bytes, flags=OOB], headers are PING: [PING: type=GET_MBRS_REQ, cluster=mockDev7, arg=leddvd0024 -27944, view_id=, is_server=false, is_coord=false, logical_name=leddvd0024 -27944, physical_addrs=54.56.30.28:50535], UDP: [channel_name=mockDev7]|
    06/01/13 23:14:10 (0.00 ms)|OOB-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|received [dst: leddvd0024 -27944, src: cacaff58-aab0-56db-4e17-0c59f0aacd6d (2 headers), size=0 bytes, flags=OOB], headers are PING: [PING: type=GET_MBRS_RSP, arg=cacaff58-aab0-56db-4e17-0c59f0aacd6d, view_id=[cacaff58-aab0-56db-4e17-0c59f0aacd6d|2], is_server=true, is_coord=true, logical_name=leddvd0024 -61221, physical_addrs=54.56.30.28:56989], UDP: [channel_name=mockDev7]|
    06/01/13 23:14:10 (0.00 ms)|OOB-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|leddvd0024 -27944: received GET_MBRS_RSP from leddvd0024 -61221: leddvd0024 -61221, view_id=[leddvd0024 -61221|2], is_server=true, is_coord=true, logical_name=leddvd0024 -61221, physical_addrs=54.56.30.28:56989|
    06/01/13 23:14:10 (0.00 ms)|[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'|VERBOSE|NONE|TEST|NONE|leddvd0024 -27944: discovery took 49 ms: responses: 2 total (1 servers (1 coord), 1 clients)|
    06/01/13 23:14:10 (0.00 ms)|[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'|VERBOSE|NONE|TEST|NONE|leddvd0024 -27944: initial_mbrs are leddvd0024 -61221 |
    06/01/13 23:14:10 (0.00 ms)|[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'|AUDITING|NONE|TEST|NONE|election results: {leddvd0024 -61221=1}|
    06/01/13 23:14:10 (0.00 ms)|[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'|AUDITING|NONE|TEST|NONE|sending JOIN(leddvd0024 -27944) to leddvd0024 -61221|
    06/01/13 23:14:10 (0.00 ms)|[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'|VERBOSE|NONE|TEST|NONE|leddvd0024 -27944: created connection to leddvd0024 -61221|
    06/01/13 23:14:10 (0.00 ms)|[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'|VERBOSE|NONE|TEST|NONE|leddvd0024 -27944 --> DATA(leddvd0024 -61221: #1, conn_id=0, first)|
    06/01/13 23:14:10 (0.00 ms)|[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'|VERBOSE|NONE|TEST|NONE|sending msg to leddvd0024 -61221, src=leddvd0024 -27944, headers are GMS: GmsHeader[JOIN_REQ]: mbr=leddvd0024 -27944, UNICAST: DATA, seqno=1, first, UDP: [channel_name=mockDev7]|
    06/01/13 23:14:10 (0.00 ms)|Incoming-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|received [dst: leddvd0024 -27944, src: leddvd0024 -61221 (2 headers), size=0 bytes], headers are UNICAST: ACK, seqno=1, UDP: [channel_name=mockDev7]|
    06/01/13 23:14:10 (0.00 ms)|Incoming-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|leddvd0024 -27944 <-- ACK(leddvd0024 -61221: #1)|
    06/01/13 23:14:10 (0.00 ms)|Incoming-2,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|received [dst: <null>, src: leddvd0024 -61221 (3 headers), size=0 bytes], headers are GMS: GmsHeader[VIEW]: view=[leddvd0024 -61221|3] [leddvd0024 -61221, 27e4bf4d-b545-4b4f-5eac-4edd4f63fed6, ecf6d73c-c0af-8ed3-0c3c-42bafe39ea20, leddvd0024 -27944], NAKACK: [MSG, seqno=3], UDP: [channel_name=mockDev7]|
    06/01/13 23:14:10 (0.00 ms)|Incoming-2,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|leddvd0024 -27944: message leddvd0024 -61221::3 was added to queue (not yet server)|
    06/01/13 23:14:11 (0.00 ms)|Timer-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|sending msg to null, src=leddvd0024 -27944, headers are PING: [PING: type=GET_MBRS_REQ, cluster=mockDev7, arg=leddvd0024 -27944, view_id=, is_server=false, is_coord=false, logical_name=leddvd0024 -27944, physical_addrs=54.56.30.28:50535], UDP: [channel_name=mockDev7]|
    06/01/13 23:14:11 (0.00 ms)|Timer-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|looping back message [dst: <null>, src: leddvd0024 -27944 (2 headers), size=0 bytes, flags=OOB]|
    06/01/13 23:14:11 (0.00 ms)|OOB-2,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|received [dst: <null>, src: leddvd0024 -27944 (2 headers), size=0 bytes, flags=OOB], headers are PING: [PING: type=GET_MBRS_REQ, cluster=mockDev7, arg=leddvd0024 -27944, view_id=, is_server=false, is_coord=false, logical_name=leddvd0024 -27944, physical_addrs=54.56.30.28:50535], UDP: [channel_name=mockDev7]|
    06/01/13 23:14:11 (0.00 ms)|OOB-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|received [dst: <null>, src: leddvd0024 -27944 (2 headers), size=0 bytes, flags=OOB], headers are PING: [PING: type=GET_MBRS_REQ, cluster=mockDev7, arg=leddvd0024 -27944, view_id=, is_server=false, is_coord=false, logical_name=leddvd0024 -27944, physical_addrs=54.56.30.28:50535], UDP: [channel_name=mockDev7]|
    06/01/13 23:14:11 (0.00 ms)|OOB-2,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|leddvd0024 -27944: received GET_MBRS_REQ from leddvd0024 -27944, sending response [PING: type=GET_MBRS_RSP, arg=leddvd0024 -27944, view_id=, is_server=false, is_coord=false, logical_name=leddvd0024 -27944, physical_addrs=54.56.30.28:50535]|
    06/01/13 23:14:11 (0.00 ms)|OOB-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|received [dst: leddvd0024 -27944, src: leddvd0024 -61221 (2 headers), size=0 bytes, flags=OOB], headers are PING: [PING: type=GET_MBRS_RSP, arg=leddvd0024 -61221, view_id=[leddvd0024 -61221|3], is_server=true, is_coord=true, logical_name=leddvd0024 -61221, physical_addrs=54.56.30.28:56989], UDP: [channel_name=mockDev7]|
    06/01/13 23:14:11 (0.00 ms)|OOB-2,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|sending msg to leddvd0024 -27944, src=leddvd0024 -27944, headers are PING: [PING: type=GET_MBRS_RSP, arg=leddvd0024 -27944, view_id=, is_server=false, is_coord=false, logical_name=leddvd0024 -27944, physical_addrs=54.56.30.28:50535], UDP: [channel_name=mockDev7]|
    06/01/13 23:14:11 (0.00 ms)|OOB-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|leddvd0024 -27944: received GET_MBRS_RSP from leddvd0024 -61221: leddvd0024 -61221, view_id=[leddvd0024 -61221|3], is_server=true, is_coord=true, logical_name=leddvd0024 -61221, physical_addrs=54.56.30.28:56989|
    06/01/13 23:14:11 (0.00 ms)|OOB-2,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|looping back message [dst: leddvd0024 -27944, src: leddvd0024 -27944 (2 headers), size=0 bytes, flags=OOB]|
    06/01/13 23:14:11 (0.00 ms)|OOB-2,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|received [dst: leddvd0024 -27944, src: leddvd0024 -27944 (2 headers), size=0 bytes, flags=OOB], headers are PING: [PING: type=GET_MBRS_RSP, arg=leddvd0024 -27944, view_id=, is_server=false, is_coord=false, logical_name=leddvd0024 -27944, physical_addrs=54.56.30.28:50535], UDP: [channel_name=mockDev7]|
    06/01/13 23:14:11 (0.00 ms)|OOB-2,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|leddvd0024 -27944: received GET_MBRS_RSP from leddvd0024 -27944: leddvd0024 -27944, view_id=, is_server=false, is_coord=false, logical_name=leddvd0024 -27944, physical_addrs=54.56.30.28:50535|
    06/01/13 23:14:12 (0.00 ms)|Incoming-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|received [dst: leddvd0024 -27944, src: leddvd0024 -61221 (3 headers), size=0 bytes], headers are GMS: GmsHeader[JOIN_RSP]: join_rsp=view: [leddvd0024 -61221|3] [leddvd0024 -61221, 27e4bf4d-b545-4b4f-5eac-4edd4f63fed6, ecf6d73c-c0af-8ed3-0c3c-42bafe39ea20, leddvd0024 -27944], digest: leddvd0024 -61221: [2 (2)], 27e4bf4d-b545-4b4f-5eac-4edd4f63fed6: [0 (0)], ecf6d73c-c0af-8ed3-0c3c-42bafe39ea20: [0 (0)], leddvd0024 -27944: [0 (0)], UNICAST: DATA, seqno=1, conn_id=4, first, UDP: [channel_name=mockDev7]|
    06/01/13 23:14:12 (0.00 ms)|Incoming-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|leddvd0024 -27944 <-- DATA(leddvd0024 -61221: #1, conn_id=4, first)|
    06/01/13 23:14:12 (0.00 ms)|Incoming-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|leddvd0024 -27944: created receiver window for leddvd0024 -61221 at seqno=#1 for conn-id=4|
    06/01/13 23:14:12 (0.00 ms)|[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'|VERBOSE|NONE|TEST|NONE|leddvd0024 -27944: JOIN-RSP=[leddvd0024 -61221|3] [leddvd0024 -61221, 27e4bf4d-b545-4b4f-5eac-4edd4f63fed6, ecf6d73c-c0af-8ed3-0c3c-42bafe39ea20, leddvd0024 -27944] [size=4]

    |
    06/01/13 23:14:12 (0.00 ms)|Incoming-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|leddvd0024 -27944 --> ACK(leddvd0024 -61221: #1)|
    06/01/13 23:14:12 (0.00 ms)|Incoming-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|sending msg to leddvd0024 -61221, src=leddvd0024 -27944, headers are UNICAST: ACK, seqno=1, UDP: [channel_name=mockDev7]|
    06/01/13 23:14:12 (0.00 ms)|[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'|AUDITING|NONE|TEST|NONE|
    [leddvd0024 -27944 setDigest()]
    existing digest: []
    new digest: leddvd0024 -61221: [2 (2)], 27e4bf4d-b545-4b4f-5eac-4edd4f63fed6: [0 (0)], ecf6d73c-c0af-8ed3-0c3c-42bafe39ea20: [0 (0)], leddvd0024 -27944: [0 (0)]
    resulting digest: leddvd0024 -27944: [0 (0)], 27e4bf4d-b545-4b4f-5eac-4edd4f63fed6: [0 (0)], ecf6d73c-c0af-8ed3-0c3c-42bafe39ea20: [0 (0)], leddvd0024 -61221: [2 (2)]|
    06/01/13 23:14:12 (0.00 ms)|[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'|AUDITING|NONE|TEST|NONE|leddvd0024 -27944: installing view [leddvd0024 -61221|3] [leddvd0024 -61221, 27e4bf4d-b545-4b4f-5eac-4edd4f63fed6, ecf6d73c-c0af-8ed3-0c3c-42bafe39ea20, leddvd0024 -27944]|
    06/01/13 23:14:12 (0.00 ms)|TransferQueueBundler,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|sending 1 msgs (61 bytes (0.1% of max_bundle_size) to 1 destination(s)|
    06/01/13 23:14:12 (0.00 ms)|[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'|AUDITING|NONE|TEST|NONE|VIEW_CHANGE received: [leddvd0024 -61221, 27e4bf4d-b545-4b4f-5eac-4edd4f63fed6, ecf6d73c-c0af-8ed3-0c3c-42bafe39ea20, leddvd0024 -27944]|
    06/01/13 23:14:12 (0.00 ms)|FD_SOCK pinger,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|sending msg to null, src=leddvd0024 -27944, headers are FD_SOCK: I_HAVE_SOCK, mbr=leddvd0024 -27944, sock_addr=54.56.30.28:48211, UDP: [channel_name=mockDev7]|
    06/01/13 23:14:12 (0.00 ms)|FD_SOCK pinger,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|looping back message [dst: <null>, src: leddvd0024 -27944 (2 headers), size=0 bytes, flags=OOB]|
    06/01/13 23:14:12 (0.00 ms)|OOB-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|received [dst: <null>, src: leddvd0024 -27944 (2 headers), size=0 bytes, flags=OOB], headers are FD_SOCK: I_HAVE_SOCK, mbr=leddvd0024 -27944, sock_addr=54.56.30.28:48211, UDP: [channel_name=mockDev7]|
    06/01/13 23:14:12 (0.00 ms)|OOB-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|i-have-sock: leddvd0024 -27944 --> 54.56.30.28:48211 (cache is {leddvd0024 -27944=54.56.30.28:48211})|
    06/01/13 23:14:12 (0.00 ms)|OOB-2,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|received [dst: <null>, src: leddvd0024 -27944 (2 headers), size=0 bytes, flags=OOB], headers are FD_SOCK: I_HAVE_SOCK, mbr=leddvd0024 -27944, sock_addr=54.56.30.28:48211, UDP: [channel_name=mockDev7]|
    06/01/13 23:14:12 (0.00 ms)|FD_SOCK pinger,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|sending msg to leddvd0024 -61221, src=leddvd0024 -27944, headers are FD_SOCK: GET_CACHE, UDP: [channel_name=mockDev7]|
    06/01/13 23:14:12 (0.00 ms)|OOB-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|received [dst: leddvd0024 -27944, src: leddvd0024 -61221 (2 headers), size=0 bytes, flags=OOB], headers are FD_SOCK: GET_CACHE_RSP, cache={leddvd0024 -27944=54.56.30.28:48211, ecf6d73c-c0af-8ed3-0c3c-42bafe39ea20=54.56.30.27:59601, 27e4bf4d-b545-4b4f-5eac-4edd4f63fed6=54.56.30.27:40775, leddvd0024 -61221=54.56.30.28:53039}, UDP: [channel_name=mockDev7]|
    06/01/13 23:14:12 (0.00 ms)|FD_SOCK pinger,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|got cache from leddvd0024 -61221: cache is {ecf6d73c-c0af-8ed3-0c3c-42bafe39ea20=54.56.30.27:59601, leddvd0024 -27944=54.56.30.28:48211, leddvd0024 -61221=54.56.30.28:53039, 27e4bf4d-b545-4b4f-5eac-4edd4f63fed6=54.56.30.27:40775}|
    06/01/13 23:14:12 (0.00 ms)|FD_SOCK pinger,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|pinger_thread started|
    06/01/13 23:14:12 (0.00 ms)|FD_SOCK pinger,mockDev7,leddvd0024 -27944|AUDITING|NONE|NONE|NONE|ping_dest is leddvd0024 -61221, pingable_mbrs=[leddvd0024 -61221, 27e4bf4d-b545-4b4f-5eac-4edd4f63fed6, ecf6d73c-c0af-8ed3-0c3c-42bafe39ea20, leddvd0024 -27944]|
    06/01/13 23:14:12 (0.00 ms)|[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'|VERBOSE|NONE|TEST|NONE|leddvd0024 -27944: resetting digest from NAKACK: leddvd0024 -27944: [0], leddvd0024 -61221: [2], ecf6d73c-c0af-8ed3-0c3c-42bafe39ea20: [0], 27e4bf4d-b545-4b4f-5eac-4edd4f63fed6: [0]|
    06/01/13 23:14:12 (0.00 ms)|[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'|AUDITING|NONE|TEST|NONE|[ergonomics] setting max_bytes to 8MB (4 members)|
    06/01/13 23:14:12 (0.00 ms)|FD_SOCK pinger,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|ping_dest=leddvd0024 -61221, ping_sock=Socket[addr=/54.56.30.28,port=53039,localport=34640], cache={ecf6d73c-c0af-8ed3-0c3c-42bafe39ea20=54.56.30.27:59601, leddvd0024 -27944=54.56.30.28:48211, leddvd0024 -61221=54.56.30.28:53039, 27e4bf4d-b545-4b4f-5eac-4edd4f63fed6=54.56.30.27:40775}|
    Broadcast service cluster name: "mockDev7", with members: "[leddvd0024 -61221, 27e4bf4d-b545-4b4f-5eac-4edd4f63fed6, ecf6d73c-c0af-8ed3-0c3c-42bafe39ea20, leddvd0024 -27944]"
    06/01/13 23:14:12 (0.00 ms)|Incoming-2,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|received [dst: leddvd0024 -27944, src: leddvd0024 -61221 (3 headers), size=0 bytes], headers are GMS: GmsHeader[JOIN_RSP]: join_rsp=view: [leddvd0024 -61221|3] [leddvd0024 -61221, 27e4bf4d-b545-4b4f-5eac-4edd4f63fed6, ecf6d73c-c0af-8ed3-0c3c-42bafe39ea20, leddvd0024 -27944], digest: leddvd0024 -61221: [2 (2)], 27e4bf4d-b545-4b4f-5eac-4edd4f63fed6: [0 (0)], ecf6d73c-c0af-8ed3-0c3c-42bafe39ea20: [0 (0)], leddvd0024 -27944: [0 (0)], UNICAST: DATA, seqno=1, conn_id=4, first, UDP: [channel_name=mockDev7]|
    06/01/13 23:14:12 (0.00 ms)|Incoming-2,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|leddvd0024 -27944 <-- DATA(leddvd0024 -61221: #1, conn_id=4, first)|
    06/01/13 23:14:12 (0.00 ms)|Incoming-2,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|leddvd0024 -27944 --> ACK(leddvd0024 -61221: #1)|
    06/01/13 23:14:12 (0.00 ms)|Incoming-2,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|sending msg to leddvd0024 -61221, src=leddvd0024 -27944, headers are UNICAST: ACK, seqno=1, UDP: [channel_name=mockDev7]|
    06/01/13 23:14:12 (0.00 ms)|[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'|VERBOSE|NONE|TEST|NONE|leddvd0024 -27944: flushing become_server_queue (1 elements)|
    06/01/13 23:14:12 (0.00 ms)|[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'|VERBOSE|NONE|TEST|NONE|leddvd0024 -27944 --> DATA(leddvd0024 -61221: #2, conn_id=0)|
    06/01/13 23:14:12 (0.00 ms)|[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'|VERBOSE|NONE|TEST|NONE|sending msg to leddvd0024 -61221, src=leddvd0024 -27944, headers are GMS: GmsHeader[VIEW_ACK]: view=null, UNICAST: DATA, seqno=2, UDP: [channel_name=mockDev7]|
    06/01/13 23:14:12 (0.00 ms)|Incoming-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|leddvd0024 -27944: received leddvd0024 -61221#3|
    06/01/13 23:14:12 (0.00 ms)|Incoming-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|leddvd0024 -27944: delivering leddvd0024 -61221#3|
    06/01/13 23:14:12 (0.00 ms)|Incoming-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|leddvd0024 -27944 --> DATA(leddvd0024 -61221: #3, conn_id=0)|
    06/01/13 23:14:12 (0.00 ms)|Incoming-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|sending msg to leddvd0024 -61221, src=leddvd0024 -27944, headers are GMS: GmsHeader[VIEW_ACK]: view=null, UNICAST: DATA, seqno=3, UDP: [channel_name=mockDev7]|
    06/01/13 23:14:12 (0.00 ms)|Incoming-2,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|received [dst: leddvd0024 -27944, src: leddvd0024 -61221 (2 headers), size=0 bytes], headers are UNICAST: ACK, seqno=2, UDP: [channel_name=mockDev7]|
    06/01/13 23:14:12 (0.00 ms)|Incoming-2,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|leddvd0024 -27944 <-- ACK(leddvd0024 -61221: #2)|
    06/01/13 23:14:12 (0.00 ms)|TransferQueueBundler,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|sending 1 msgs (61 bytes (0.1% of max_bundle_size) to 1 destination(s)|
    06/01/13 23:14:12 (0.00 ms)|Incoming-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|received [dst: leddvd0024 -27944, src: leddvd0024 -61221 (2 headers), size=0 bytes], headers are UNICAST: ACK, seqno=3, UDP: [channel_name=mockDev7]|
    06/01/13 23:14:12 (0.00 ms)|Incoming-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|leddvd0024 -27944 <-- ACK(leddvd0024 -61221: #3)|

    06/01/13 23:14:15 (-111.00 ms)|Timer-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|leddvd0024 -27944: discovery took 3884 ms: responses: 2 total (1 servers (1 coord), 1 clients)|
    06/01/13 23:14:15 (-111.00 ms)|Timer-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|Discovery results:
    [leddvd0024 -61221]: view_id=[leddvd0024 -61221|3]
    |
    06/01/13 23:14:17 (-111.00 ms)|OOB-2,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|received [dst: <null>, src: leddvd0024 -61221 (2 headers), size=0 bytes, flags=OOB], headers are PING: [PING: type=GET_MBRS_REQ, cluster=mockDev7, view_id=[leddvd0024 -61221|3]], UDP: [channel_name=mockDev7]|
    06/01/13 23:14:17 (-111.00 ms)|Timer-2,mockDev7,leddvd0024 -27944|AUDITING|NONE|NONE|NONE|leddvd0024 -27944: sending are-you-alive msg to leddvd0024 -61221|
    06/01/13 23:14:17 (-111.00 ms)|Timer-2,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|sending msg to leddvd0024 -61221, src=leddvd0024 -27944, headers are FD: heartbeat, UDP: [channel_name=mockDev7]|
    06/01/13 23:14:17 (-111.00 ms)|OOB-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|received [dst: leddvd0024 -27944, src: leddvd0024 -61221 (2 headers), size=0 bytes, flags=OOB], headers are FD: heartbeat ack, UDP: [channel_name=mockDev7]|
    06/01/13 23:14:20 (-111.00 ms)|Timer-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|leddvd0024 -27944: setting latest_local_digest from NAKACK: leddvd0024 -27944: [0], leddvd0024 -61221: [3], ecf6d73c-c0af-8ed3-0c3c-42bafe39ea20: [0], 27e4bf4d-b545-4b4f-5eac-4edd4f63fed6: [0]|
    06/01/13 23:14:20 (-111.00 ms)|Timer-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|leddvd0024 -27944: sending stable msg leddvd0024 -27944: [0], leddvd0024 -61221: [3], ecf6d73c-c0af-8ed3-0c3c-42bafe39ea20: [0], 27e4bf4d-b545-4b4f-5eac-4edd4f63fed6: [0]|
    06/01/13 23:14:20 (-111.00 ms)|Timer-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|sending msg to null, src=leddvd0024 -27944, headers are STABLE: [STABLE_GOSSIP]: digest is leddvd0024 -27944: [0 (0)], 27e4bf4d-b545-4b4f-5eac-4edd4f63fed6: [0 (0)], ecf6d73c-c0af-8ed3-0c3c-42bafe39ea20: [0 (0)], leddvd0024 -61221: [3 (3)], UDP: [channel_name=mockDev7]|
    06/01/13 23:14:20 (-111.00 ms)|Timer-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|looping back message [dst: <null>, src: leddvd0024 -27944 (2 headers), size=0 bytes, flags=OOB|NO_RELIABILITY]|
    06/01/13 23:14:20 (-111.00 ms)|OOB-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|received [dst: <null>, src: leddvd0024 -27944 (2 headers), size=0 bytes, flags=OOB|NO_RELIABILITY], headers are STABLE: [STABLE_GOSSIP]: digest is leddvd0024 -27944: [0 (0)], 27e4bf4d-b545-4b4f-5eac-4edd4f63fed6: [0 (0)], ecf6d73c-c0af-8ed3-0c3c-42bafe39ea20: [0 (0)], leddvd0024 -61221: [3 (3)], UDP: [channel_name=mockDev7]|
    06/01/13 23:14:20 (-111.00 ms)|OOB-2,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|received [dst: <null>, src: leddvd0024 -27944 (2 headers), size=0 bytes, flags=OOB|NO_RELIABILITY], headers are STABLE: [STABLE_GOSSIP]: digest is leddvd0024 -27944: [0 (0)], 27e4bf4d-b545-4b4f-5eac-4edd4f63fed6: [0 (0)], ecf6d73c-c0af-8ed3-0c3c-42bafe39ea20: [0 (0)], leddvd0024 -61221: [3 (3)], UDP: [channel_name=mockDev7]|
    06/01/13 23:14:20 (-111.00 ms)|OOB-2,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|leddvd0024 -27944: handling digest from leddvd0024 -27944 (0 votes):
    mine: leddvd0024 -27944: [0], leddvd0024 -61221: [2], ecf6d73c-c0af-8ed3-0c3c-42bafe39ea20: [0], 27e4bf4d-b545-4b4f-5eac-4edd4f63fed6: [0]
    other: leddvd0024 -27944: [0], leddvd0024 -61221: [3], ecf6d73c-c0af-8ed3-0c3c-42bafe39ea20: [0], 27e4bf4d-b545-4b4f-5eac-4edd4f63fed6: [0]
    result: leddvd0024 -27944: [0], leddvd0024 -61221: [2], ecf6d73c-c0af-8ed3-0c3c-42bafe39ea20: [0], 27e4bf4d-b545-4b4f-5eac-4edd4f63fed6: [0]
    |
    06/01/13 23:14:22 (-111.00 ms)|Timer-3,mockDev7,leddvd0024 -27944|AUDITING|NONE|NONE|NONE|leddvd0024 -27944: sending are-you-alive msg to leddvd0024 -61221|
    06/01/13 23:14:22 (-111.00 ms)|Timer-3,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|sending msg to leddvd0024 -61221, src=leddvd0024 -27944, headers are FD: heartbeat, UDP: [channel_name=mockDev7]|
    06/01/13 23:14:22 (-111.00 ms)|OOB-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|received [dst: leddvd0024 -27944, src: leddvd0024 -61221 (2 headers), size=0 bytes, flags=OOB], headers are FD: heartbeat ack, UDP: [channel_name=mockDev7]|
    06/01/13 23:14:25 (-111.00 ms)|OOB-2,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|received [dst: <null>, src: leddvd0024 -61221 (2 headers), size=0 bytes, flags=OOB], headers are PING: [PING: type=GET_MBRS_REQ, cluster=mockDev7, view_id=[leddvd0024 -61221|3]], UDP: [channel_name=mockDev7]|
    06/01/13 23:14:27 (-111.00 ms)|Timer-3,mockDev7,leddvd0024 -27944|AUDITING|NONE|NONE|NONE|leddvd0024 -27944: sending are-you-alive msg to leddvd0024 -61221|
    06/01/13 23:14:27 (-111.00 ms)|Timer-3,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|sending msg to leddvd0024 -61221, src=leddvd0024 -27944, headers are FD: heartbeat, UDP: [channel_name=mockDev7]|
    06/01/13 23:14:27 (-111.00 ms)|OOB-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|received [dst: leddvd0024 -27944, src: leddvd0024 -61221 (2 headers), size=0 bytes, flags=OOB], headers are FD: heartbeat ack, UDP: [channel_name=mockDev7]|
    06/01/13 23:14:32 (-111.00 ms)|Timer-3,mockDev7,leddvd0024 -27944|AUDITING|NONE|NONE|NONE|leddvd0024 -27944: sending are-you-alive msg to leddvd0024 -61221|
    06/01/13 23:14:32 (-111.00 ms)|Timer-3,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|sending msg to leddvd0024 -61221, src=leddvd0024 -27944, headers are FD: heartbeat, UDP: [channel_name=mockDev7]|
    06/01/13 23:14:32 (-111.00 ms)|OOB-2,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|received [dst: leddvd0024 -27944, src: leddvd0024 -61221 (2 headers), size=0 bytes, flags=OOB], headers are FD: heartbeat ack, UDP: [channel_name=mockDev7]|
    06/01/13 23:14:36 (-111.00 ms)|OOB-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|received [dst: <null>, src: leddvd0024 -61221 (2 headers), size=0 bytes, flags=OOB], headers are PING: [PING: type=GET_MBRS_REQ, cluster=mockDev7, view_id=[leddvd0024 -61221|3]], UDP: [channel_name=mockDev7]|
    06/01/13 23:14:37 (-111.00 ms)|Timer-3,mockDev7,leddvd0024 -27944|AUDITING|NONE|NONE|NONE|leddvd0024 -27944: sending are-you-alive msg to leddvd0024 -61221|
    06/01/13 23:14:37 (-111.00 ms)|Timer-3,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|sending msg to leddvd0024 -61221, src=leddvd0024 -27944, headers are FD: heartbeat, UDP: [channel_name=mockDev7]|
    06/01/13 23:14:37 (-111.00 ms)|OOB-2,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|received [dst: leddvd0024 -27944, src: leddvd0024 -61221 (2 headers), size=0 bytes, flags=OOB], headers are FD: heartbeat ack, UDP: [channel_name=mockDev7]|
    06/01/13 23:14:42 (-111.00 ms)|Timer-1,mockDev7,leddvd0024 -27944|AUDITING|NONE|NONE|NONE|leddvd0024 -27944: sending are-you-alive msg to leddvd0024 -61221|
    06/01/13 23:14:42 (-111.00 ms)|Timer-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|sending msg to leddvd0024 -61221, src=leddvd0024 -27944, headers are FD: heartbeat, UDP: [channel_name=mockDev7]|
    06/01/13 23:14:42 (-111.00 ms)|OOB-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|received [dst: leddvd0024 -27944, src: leddvd0024 -61221 (2 headers), size=0 bytes, flags=OOB], headers are FD: heartbeat ack, UDP: [channel_name=mockDev7]|
    06/01/13 23:14:43 (-111.00 ms)|OOB-2,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|received [dst: <null>, src: leddvd0024 -61221 (2 headers), size=0 bytes, flags=OOB], headers are PING: [PING: type=GET_MBRS_REQ, cluster=mockDev7, view_id=[leddvd0024 -61221|3]], UDP: [channel_name=mockDev7]|
    06/01/13 23:14:47 (-111.00 ms)|Timer-3,mockDev7,leddvd0024 -27944|AUDITING|NONE|NONE|NONE|leddvd0024 -27944: sending are-you-alive msg to leddvd0024 -61221|
    06/01/13 23:14:47 (-111.00 ms)|Timer-3,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|sending msg to leddvd0024 -61221, src=leddvd0024 -27944, headers are FD: heartbeat, UDP: [channel_name=mockDev7]|
    06/01/13 23:14:47 (-111.00 ms)|OOB-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|received [dst: leddvd0024 -27944, src: leddvd0024 -61221 (2 headers), size=0 bytes, flags=OOB], headers are FD: heartbeat ack, UDP: [channel_name=mockDev7]|
    06/01/13 23:14:52 (-111.00 ms)|Timer-3,mockDev7,leddvd0024 -27944|AUDITING|NONE|NONE|NONE|leddvd0024 -27944: sending are-you-alive msg to leddvd0024 -61221|
    06/01/13 23:14:52 (-111.00 ms)|Timer-3,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|sending msg to leddvd0024 -61221, src=leddvd0024 -27944, headers are FD: heartbeat, UDP: [channel_name=mockDev7]|
    06/01/13 23:14:52 (-111.00 ms)|OOB-2,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|received [dst: leddvd0024 -27944, src: leddvd0024 -61221 (2 headers), size=0 bytes, flags=OOB], headers are FD: heartbeat ack, UDP: [channel_name=mockDev7]|
    06/01/13 23:14:54 (-111.00 ms)|OOB-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|received [dst: <null>, src: leddvd0024 -61221 (2 headers), size=0 bytes, flags=OOB], headers are PING: [PING: type=GET_MBRS_REQ, cluster=mockDev7, view_id=[leddvd0024 -61221|3]], UDP: [channel_name=mockDev7]|
    06/01/13 23:14:57 (-111.00 ms)|Timer-2,mockDev7,leddvd0024 -27944|AUDITING|NONE|NONE|NONE|leddvd0024 -27944: sending are-you-alive msg to leddvd0024 -61221|
    06/01/13 23:14:57 (-111.00 ms)|Timer-2,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|sending msg to leddvd0024 -61221, src=leddvd0024 -27944, headers are FD: heartbeat, UDP: [channel_name=mockDev7]|
    06/01/13 23:14:57 (-111.00 ms)|OOB-2,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|received [dst: leddvd0024 -27944, src: leddvd0024 -61221 (2 headers), size=0 bytes, flags=OOB], headers are FD: heartbeat ack, UDP: [channel_name=mockDev7]|
    **Using CMnWeblogicallableStatement
    06/01/13 23:15:01 (-111.00 ms)|OOB-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|received [dst: <null>, src: leddvd0024 -61221 (2 headers), size=0 bytes, flags=OOB], headers are PING: [PING: type=GET_MBRS_REQ, cluster=mockDev7, view_id=[leddvd0024 -61221|3]], UDP: [channel_name=mockDev7]|
    06/01/13 23:15:06 (-111.00 ms)|Timer-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|leddvd0024 -27944: setting latest_local_digest from NAKACK: leddvd0024 -27944: [0], leddvd0024 -61221: [3], ecf6d73c-c0af-8ed3-0c3c-42bafe39ea20: [0], 27e4bf4d-b545-4b4f-5eac-4edd4f63fed6: [0]|
    06/01/13 23:15:06 (-111.00 ms)|Timer-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|leddvd0024 -27944: sending stable msg leddvd0024 -27944: [0], leddvd0024 -61221: [3], ecf6d73c-c0af-8ed3-0c3c-42bafe39ea20: [0], 27e4bf4d-b545-4b4f-5eac-4edd4f63fed6: [0]|
    06/01/13 23:15:06 (-111.00 ms)|Timer-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|sending msg to null, src=leddvd0024 -27944, headers are STABLE: [STABLE_GOSSIP]: digest is leddvd0024 -27944: [0 (0)], 27e4bf4d-b545-4b4f-5eac-4edd4f63fed6: [0 (0)], ecf6d73c-c0af-8ed3-0c3c-42bafe39ea20: [0 (0)], leddvd0024 -61221: [3 (3)], UDP: [channel_name=mockDev7]|
    06/01/13 23:15:06 (-111.00 ms)|Timer-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|looping back message [dst: <null>, src: leddvd0024 -27944 (2 headers), size=0 bytes, flags=OOB|NO_RELIABILITY]|
    06/01/13 23:15:06 (-111.00 ms)|OOB-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|received [dst: <null>, src: leddvd0024 -27944 (2 headers), size=0 bytes, flags=OOB|NO_RELIABILITY], headers are STABLE: [STABLE_GOSSIP]: digest is leddvd0024 -27944: [0 (0)], 27e4bf4d-b545-4b4f-5eac-4edd4f63fed6: [0 (0)], ecf6d73c-c0af-8ed3-0c3c-42bafe39ea20: [0 (0)], leddvd0024 -61221: [3 (3)], UDP: [channel_name=mockDev7]|
    06/01/13 23:15:06 (-111.00 ms)|OOB-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|received [dst: <null>, src: leddvd0024 -27944 (2 headers), size=0 bytes, flags=OOB|NO_RELIABILITY], headers are STABLE: [STABLE_GOSSIP]: digest is leddvd0024 -27944: [0 (0)], 27e4bf4d-b545-4b4f-5eac-4edd4f63fed6: [0 (0)], ecf6d73c-c0af-8ed3-0c3c-42bafe39ea20: [0 (0)], leddvd0024 -61221: [3 (3)], UDP: [channel_name=mockDev7]|
    06/01/13 23:15:07 (-111.00 ms)|Timer-3,mockDev7,leddvd0024 -27944|AUDITING|NONE|NONE|NONE|leddvd0024 -27944: sending are-you-alive msg to leddvd0024 -61221|
    06/01/13 23:15:07 (-111.00 ms)|Timer-3,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|sending msg to leddvd0024 -61221, src=leddvd0024 -27944, headers are FD: heartbeat, UDP: [channel_name=mockDev7]|
    06/01/13 23:15:07 (-111.00 ms)|OOB-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|received [dst: leddvd0024 -27944, src: leddvd0024 -61221 (2 headers), size=0 bytes, flags=OOB], headers are FD: heartbeat ack, UDP: [channel_name=mockDev7]|
    <Jun 1,="" 2013="" 11:15:08="" PM="" EDT=""> <Notice> <WebLogicServer> <BEA-000365> <Server state="" changed="" to="" RUNNING="">
    <Jun 1,="" 2013="" 11:15:08="" PM="" EDT=""> <Notice> <WebLogicServer> <BEA-000360> <Server started="" in="" RUNNING="" mode="">
    06/01/13 23:15:11 (-111.00 ms)|OOB-2,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|received [dst: <null>, src: leddvd0024 -61221 (2 headers), size=0 bytes, flags=OOB], headers are PING: [PING: type=GET_MBRS_REQ, cluster=mockDev7, view_id=[leddvd0024 -61221|3]], UDP: [channel_name=mockDev7]|
    06/01/13 23:15:12 (-111.00 ms)|Timer-2,mockDev7,leddvd0024 -27944|AUDITING|NONE|NONE|NONE|leddvd0024 -27944: sending are-you-alive msg to leddvd0024 -61221|
    06/01/13 23:15:12 (-111.00 ms)|Timer-2,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|sending msg to leddvd0024 -61221, src=leddvd0024 -27944, headers are FD: heartbeat, UDP: [channel_name=mockDev7]|
    06/01/13 23:15:12 (-111.00 ms)|OOB-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|received [dst: leddvd0024 -27944, src: leddvd0024 -61221 (2 headers), size=0 bytes, flags=OOB], headers are FD: heartbeat ack, UDP: [channel_name=mockDev7]|
    06/01/13 23:15:17 (-111.00 ms)|Timer-1,mockDev7,leddvd0024 -27944|AUDITING|NONE|NONE|NONE|leddvd0024 -27944: sending are-you-alive msg to leddvd0024 -61221|
    06/01/13 23:15:17 (-111.00 ms)|Timer-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|sending msg to leddvd0024 -61221, src=leddvd0024 -27944, headers are FD: heartbeat, UDP: [channel_name=mockDev7]|
    06/01/13 23:15:17 (-111.00 ms)|OOB-2,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|received [dst: leddvd0024 -27944, src: leddvd0024 -61221 (2 headers), size=0 bytes, flags=OOB], headers are FD: heartbeat ack, UDP: [channel_name=mockDev7]|
    06/01/13 23:15:19 (-111.00 ms)|Timer-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|leddvd0024 -27944: setting latest_local_digest from NAKACK: leddvd0024 -27944: [0], leddvd0024 -61221: [3], ecf6d73c-c0af-8ed3-0c3c-42bafe39ea20: [0], 27e4bf4d-b545-4b4f-5eac-4edd4f63fed6: [0]|
    06/01/13 23:15:19 (-111.00 ms)|Timer-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|leddvd0024 -27944: sending stable msg leddvd0024 -27944: [0], leddvd0024 -61221: [3], ecf6d73c-c0af-8ed3-0c3c-42bafe39ea20: [0], 27e4bf4d-b545-4b4f-5eac-4edd4f63fed6: [0]|
    06/01/13 23:15:19 (-111.00 ms)|Timer-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|sending msg to null, src=leddvd0024 -27944, headers are STABLE: [STABLE_GOSSIP]: digest is leddvd0024 -27944: [0 (0)], 27e4bf4d-b545-4b4f-5eac-4edd4f63fed6: [0 (0)], ecf6d73c-c0af-8ed3-0c3c-42bafe39ea20: [0 (0)], leddvd0024 -61221: [3 (3)], UDP: [channel_name=mockDev7]|
    06/01/13 23:15:19 (-111.00 ms)|Timer-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|looping back message [dst: <null>, src: leddvd0024 -27944 (2 headers), size=0 bytes, flags=OOB|NO_RELIABILITY]|
    06/01/13 23:15:19 (-111.00 ms)|OOB-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|received [dst: <null>, src: leddvd0024 -27944 (2 headers), size=0 bytes, flags=OOB|NO_RELIABILITY], headers are STABLE: [STABLE_GOSSIP]: digest is leddvd0024 -27944: [0 (0)], 27e4bf4d-b545-4b4f-5eac-4edd4f63fed6: [0 (0)], ecf6d73c-c0af-8ed3-0c3c-42bafe39ea20: [0 (0)], leddvd0024 -61221: [3 (3)], UDP: [channel_name=mockDev7]|
    06/01/13 23:15:19 (-111.00 ms)|OOB-2,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|received [dst: <null>, src: leddvd0024 -27944 (2 headers), size=0 bytes, flags=OOB|NO_RELIABILITY], headers are STABLE: [STABLE_GOSSIP]: digest is leddvd0024 -27944: [0 (0)], 27e4bf4d-b545-4b4f-5eac-4edd4f63fed6: [0 (0)], ecf6d73c-c0af-8ed3-0c3c-42bafe39ea20: [0 (0)], leddvd0024 -61221: [3 (3)], UDP: [channel_name=mockDev7]|
    06/01/13 23:15:20 (-111.00 ms)|OOB-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|received [dst: <null>, src: leddvd0024 -61221 (2 headers), size=0 bytes, flags=OOB], headers are PING: [PING: type=GET_MBRS_REQ, cluster=mockDev7, view_id=[leddvd0024 -61221|3]], UDP: [channel_name=mockDev7]|
    06/01/13 23:15:22 (-111.00 ms)|Timer-1,mockDev7,leddvd0024 -27944|AUDITING|NONE|NONE|NONE|leddvd0024 -27944: sending are-you-alive msg to leddvd0024 -61221|
    06/01/13 23:15:22 (-111.00 ms)|Timer-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|sending msg to leddvd0024 -61221, src=leddvd0024 -27944, headers are FD: heartbeat, UDP: [channel_name=mockDev7]|
    06/01/13 23:15:22 (-111.00 ms)|OOB-2,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|received [dst: leddvd0024 -27944, src: leddvd0024 -61221 (2 headers), size=0 bytes, flags=OOB], headers are FD: heartbeat ack, UDP: [channel_name=mockDev7]|
    06/01/13 23:15:23 (-111.00 ms)|OOB-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|received [dst: <null>, src: leddvd0024 -61221 (2 headers), size=0 bytes, flags=OOB|NO_RELIABILITY], headers are STABLE: [STABLE_GOSSIP]: digest is leddvd0024 -27944: [0 (0)], 27e4bf4d-b545-4b4f-5eac-4edd4f63fed6: [0 (0)], ecf6d73c-c0af-8ed3-0c3c-42bafe39ea20: [0 (0)], leddvd0024 -61221: [3 (3)], UDP: [channel_name=mockDev7]|
    06/01/13 23:15:23 (-111.00 ms)|OOB-1,mockDev7,leddvd0024 -27944|VERBOSE|NONE|NONE|NONE|leddvd0024 -27944: handling digest from leddvd0024 -61221 (1 votes):
    mine: leddvd0024 -27944: [0], leddvd0024 -61221: [2], ecf6d73c-c0af-8ed3-0c3c-42bafe39ea20: [0], 27e4bf4d-b545-4b4f-5eac-4edd4f63fed6: [0]
    other: leddvd0024 -27944: [0], leddvd0024 -61221: [3], ecf6d73c-c0af-8ed3-0c3c-42bafe39ea20: [0], 27e4bf4d-b545-4b4f-5eac-4edd4f63fed6: [0]
    result: leddvd0024 -27944: [0], leddvd0024 -61221: [2], ecf6d73c-c0af-8ed3-0c3c-42bafe39ea20: [0], 27e4bf4d-b545-4b4f-5eac-4edd4f63fed6: [0]
    |

    Also note
    node2 sees node1, node2, node3
    node4 sees 27e4bf4d-b545-4b4f-5eac-4edd4f63fed6,
    ecf6d73c-c0af-8ed3-0c3c-42bafe39ea20, node3, node4
    node1 sees node1, node2, node3
    node3 sees node1, node2, node3, node4

    Also make a note if i start all my nodes the cluster is successfully formed with out any issue.

    Any help would be greatly appreciated.

    Regards
    Rajan

     
    Last edit: Rajan Krishnan 2013-06-02
  • Rajan Krishnan

    Rajan Krishnan - 2013-06-02

    Just to add when in perform ping on my multicast i see this response, if Ping is disabled will be Discovery through PING will fail and that is the reason for my above cluster formation.. But if that is the case how does when all the nodes start at the same time work, they never use PING

    -sh-3.2$ ping 239.56.30.2
    PING 239.56.30.2 (239.56.30.2) 56(84) bytes of data.

    --- 239.56.30.2 ping statistics ---
    11 packets transmitted, 0 received, 100% packet loss, time 10026ms

    You have new mail in /var/spool/mail/

     

Log in to post a comment.

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:





No, thanks