[javagroups-users] Help understanding why TCP cluster is not merging after islanded node

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 454-5900

Hey Bela, I am back again with some more failure scenarios that for some
reason aren't quite working properly.  This is a 9 node TCP cluster
where one member I 'cut off' via ip table rules (drop all IN/OUT tcp
packets to 7800 (normal channel) and 7804 (FD_SOCK)) ports.  I get the
system into a state where eventually the islanded node is truly in a
group of its own and the rest see everyone except him.  So far so good
with FD, etc.  Then I undo the iptables change to allow 7800 and 7804
once again.  I wait for a very, very long time (10+ minutes) and it
never rejoins the main cluster.  I see MERGE2 results as follows which
is strange to me - seems that there should be a mix of two views in the
results??

>From the coordinator of the main cluster (asmblade35 is the islanded
node):

2012-06-11 16:25:08,325
[Timer-3,135.9.95.43_InterCluster-3.0,cmarch2-10072]
jgroups.protocols.TCPPING TRACE - discovery took 8004 ms: responses: 9
total (9 servers (1 coord), 0 clients)

2012-06-11 16:25:08,325
[Timer-3,135.9.95.43_InterCluster-3.0,cmarch2-10072]
jgroups.protocols.MERGE2 TRACE - Discovery results:

[defarch6-13807]: view_id=[asmblade35-16171|0] ([asmblade35-16171|0]
[asmblade35-16171])

[cmarch2-10072]: view_id=[asmblade35-16171|0] ([asmblade35-16171|0]
[asmblade35-16171])

[asmblade35-16171]: view_id=[asmblade35-16171|0] ([asmblade35-16171|0]
[asmblade35-16171])

[cmarch74-11534]: view_id=[asmblade35-16171|0] ([asmblade35-16171|0]
[asmblade35-16171])

[cmarch70-30275]: view_id=[asmblade35-16171|0] ([asmblade35-16171|0]
[asmblade35-16171])

[asmblade34-1530]: view_id=[asmblade35-16171|0] ([asmblade35-16171|0]
[asmblade35-16171])

[cmarch80-42927]: view_id=[asmblade35-16171|0] ([asmblade35-16171|0]
[asmblade35-16171])

[cmarch72-18235]: view_id=[asmblade35-16171|0] ([asmblade35-16171|0]
[asmblade35-16171])

[cmarch76-46755]: view_id=[asmblade35-16171|0] ([asmblade35-16171|0]
[asmblade35-16171])

>From the islanded node, asmblade35, I see the main cluster:

2012-06-11 16:25:57,356
[Timer-3,135.9.95.43_InterCluster-3.0,asmblade35-46813]
jgroups.protocols.TCPPING TRACE - discovery took 8002 ms: responses: 12
total (12 servers (1 coord), 0 clients)

2012-06-11 16:25:57,357
[Timer-3,135.9.95.43_InterCluster-3.0,asmblade35-46813]
jgroups.protocols.MERGE2 TRACE - Discovery results:

[asmblade34-63376]: view_id=[cmarch2-10072|178]

[defarch6-13807]: view_id=[cmarch2-10072|178]

[asmblade35-36778]: view_id=[cmarch2-10072|178]

[asmblade35-16171]: view_id=[cmarch2-10072|178]

[asmblade35-46813]: view_id=[cmarch2-10072|178]

[cmarch72-18235]: view_id=[cmarch2-10072|178]

[cmarch80-42927]: view_id=[cmarch2-10072|178]

[cmarch74-11534]: view_id=[cmarch2-10072|178]

[cmarch76-46755]: view_id=[cmarch2-10072|178]

[asmblade34-1530]: view_id=[cmarch2-10072|178]

[cmarch2-10072]: view_id=[cmarch2-10072|178]

[cmarch70-30275]: view_id=[cmarch2-10072|178]

Should the view_ids in the above be the IDs from their own cluster's
viewpoint??  Either way, the MERGE2 seems to hit this code below and
stop processing

List<View> different_views=detectDifferentViews(views);

if(different_views.size() <= 1) {

num_inconsistent_views=0;

      return;

}

I added a print of 'different_views' and see that it produces just a
list of size one - with the above code, it returns and the cluster
remains split in two:

2012-06-11 17:14:31,271
[Timer-5,135.9.95.43_InterCluster-3.0,asmblade35-26234]
jgroups.protocols.MERGE2 INFO - KEN: Detect diff views: (1)
[[cmarch70-30275|186] [cmarch70-30275, cmarch74-11534, cmarch72-18235,
cmarch80-42927, cmarch76-46755, asmblade34-1530, cmarch2-36776,
defarch6-9402]]

I ensured that all the hosts are in the initial_hosts list (including
itself).  Here is the stack; any conflicting issues with the times?

<config xmlns="urn:org:jgroups"

        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

        xsi:schemaLocation="urn:org:jgroups
http://www.jgroups.org/schema/JGroups-3.0.xsd">

    <TCP bind_port="7800"

         loopback="false"

         recv_buf_size="${tcp.recv_buf_size:20M}"

         send_buf_size="${tcp.send_buf_size:640K}"

         discard_incompatible_packets="true"

         max_bundle_size="64K"

         max_bundle_timeout="30"

         enable_bundling="true"

         use_send_queues="false"

         sock_conn_timeout="1500"

         timer_type="new"

         timer.min_threads="6"

         timer.max_threads="10"

         timer.keep_alive_time="3000"

         timer.queue_max_size="500"

         thread_pool.enabled="true"

         thread_pool.min_threads="1"

         thread_pool.max_threads="10"

         thread_pool.keep_alive_time="5000"

         thread_pool.queue_enabled="false"

         thread_pool.queue_max_size="10000"

         thread_pool.rejection_policy="run"

         oob_thread_pool.enabled="true"

         oob_thread_pool.min_threads="2"

         oob_thread_pool.max_threads="20"

         oob_thread_pool.keep_alive_time="5000"

         oob_thread_pool.queue_enabled="false"

         oob_thread_pool.queue_max_size="100"

         oob_thread_pool.rejection_policy="discard"/>

        <TCPPING timeout="8000"

initial_hosts="${jgroups.tcpping.initial_hosts:localhost[7800],localhost
[7801]}"

             port_range="0"

             num_initial_members="5"

             return_entire_cache="true"/>

    <ENCRYPT key_store_name="${jgroups.key_store_name}"

             encrypt_entire_message="true"

             store_password="${jgroups.store_password}"

             key_password="${jgroups.key_password}"

             alias="${jgroups.alias}" />

    <MERGE2  max_interval="100000"

             min_interval="20000"/>

    <FD_SOCK start_port="7804" />

    <FD timeout="10000" max_tries="5" />

    <VERIFY_SUSPECT timeout="10000"  />

    <BARRIER />

    <pbcast.NAKACK use_mcast_xmit="false"

                   exponential_backoff="500"

                   discard_delivered_msgs="true"/>

    <UNICAST2 />

    <pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000"

                   max_bytes="4M"/>

    <pbcast.GMS print_local_addr="true"

                join_timeout="10000"

                leave_timeout="10000"

                merge_timeout="30000"

                view_ack_collection_timeout="6000"

                view_bundling="true"

                max_bundling_time="1000"/>

    <UFC max_credits="2M"

         min_threshold="0.4"

max_block_times="500:2,1500:5,5000:50,20000:200,100000:500,1000000:1000"
/>

    <MFC max_credits="2M"

         min_threshold="0.4"/>

    <FRAG2 frag_size="60K"  />

    <pbcast.STATE_TRANSFER/>

</config>

Thanks again!

Ken