JGroups

Brian Stansberry wrote:
> One category of errors JGroups users see is improperly configured
> multicast on the interface the transport is using.

What do you mean by that ? A sender should always receive a multicast 
packet sent through interface X unless X is disconnected, or IP 
multicasting is disabled.

> Could the UDP transport help here? Perhaps send a multicast message
> during the channel connect phase, and block a short period on a future
> waiting to receive your own message. If the message isn't received, log
> an ERROR with a nice helpful diagnostic.

I can do that but the issue is that (a) this will add to the startup 
time, (b) if no response is received, would you want to abort channel 
creation and (c) what if the UDP packet is simply lost (send multiple 
packets ?).

If I do this, we'd have to be able to turn this off, and that would be 
the default.

> I bring this up because JBoss AS changed to loopback="true" in its stack
> configs solely because users with network config issues were reporting
> cryptic FLUSH failures when loopback="false". FLUSH would fail due to
> inability to receive its own messages. Turning on loopback allowed FLUSH
> to work, but of course nodes wouldn't properly cluster. I saw failing to
> cluster as a better behavior, as it's something that's well covered in
> various wiki pages, docs, etc. But even better would be failing to
> cluster with a nice ERROR message in the logs explaining that multicast
> wasn't working properly on interface X.

Can you create a JIRA issue for this ?

-- 
Bela Ban
Lead JGroups / Clustering Team
JBoss - a division of Red Hat

Brian Stansberry-4 wrote:
> 
> One category of errors JGroups users see is improperly configured
> multicast on the interface the transport is using.
> 
> Could the UDP transport help here? Perhaps send a multicast message during
> the channel connect phase, and block a short period on a future waiting to
> receive your own message.  If the message isn't received, log an ERROR
> with a nice helpful diagnostic.
> 

Yeah, exactly right.  We've had tons of issues with multicast routing. 
Anyone setting up a mature networking infrastructure should first, IMO,
worry about multicast connectivity.  For example, Cisco firewall settings
through our core routers had a couple of multicast obtuse settings that
we've had to massage.  We recently moved data centers hot/hot and had a ton
of multicast tunneling issues.  We've had to log 3-4 bugs with Cisco as
we've seen complete and partial multicast failures when our configurations
seemed to be good.  We've gotten a bunch of "just reboot it" answers which
have worked and not helped our risk analysis numbers.

After a particularly bad service failure we decided to add multicast
connectivity testing to the VIEW_SYNC protocol since it is already
multicasting around the views.  We have a JMX button which sends out a
multicast test view and expects everyone to ACK it.  It can sleep and report
back on the number of misses.

We also talked [a bit] about some sort of connection testing to stop a node
from joining the network unless it can prove they have good full-duplex
multicast.  A node can actually join a channel without it as long as the
coordinator is ok.  However, it will immediately get behind on the views and
can (I think) freeze other nodes from joining since the new views are never
ACK'd.  This really horks the coordinator.

gray

-- 
View this message in context: http://www.nabble.com/Multicast-validation-by-UDP-tp20386911p20399341.html
Sent from the JGroups - Dev mailing list archive at Nabble.com.

Bela Ban wrote:
> 
> 
> Brian Stansberry wrote:
>> One category of errors JGroups users see is improperly configured
>> multicast on the interface the transport is using.
> 
> What do you mean by that ? A sender should always receive a multicast 
> packet sent through interface X unless X is disconnected, or IP 
> multicasting is disabled.
> 

I'll have to dig through the jboss-dev list archive to find the exact 
situation referred to in my last paragraph. It was something where 
multicast wasn't working properly on lo on one of Adrian Brock's 
machines, probably no multicast route or something. Same thing happened 
on one of the hudson slaves.

>> Could the UDP transport help here? Perhaps send a multicast message
>> during the channel connect phase, and block a short period on a future
>> waiting to receive your own message. If the message isn't received, log
>> an ERROR with a nice helpful diagnostic.
> 
> I can do that but the issue is that (a) this will add to the startup 
> time, 

I wouldn't think it would add much though in the standard case where the 
OS passes the multicast packet right back.

(b) if no response is received, would you want to abort channel
> creation and 

No. Others might have different opinion (e.g. gray's comment on this 
thread where he's looking for something more sophisticated) but all I 
was looking for was a nice ERROR message in the log, so if things go 
wrong, people can see it right at the beginning.

(c) what if the UDP packet is simply lost (send multiple
> packets ?).
> 

I'd say no, since in that case all we'd be doing is logging a spurious 
ERROR message. I wouldn't think the odds of not receiving your own 
multicast packet would be high.

> If I do this, we'd have to be able to turn this off, and that would be 
> the default.
> 

No problem. Probably set a timeout for how long to wait with a default 
that disables this.

>> I bring this up because JBoss AS changed to loopback="true" in its stack
>> configs solely because users with network config issues were reporting
>> cryptic FLUSH failures when loopback="false". FLUSH would fail due to
>> inability to receive its own messages. Turning on loopback allowed FLUSH
>> to work, but of course nodes wouldn't properly cluster. I saw failing to
>> cluster as a better behavior, as it's something that's well covered in
>> various wiki pages, docs, etc. But even better would be failing to
>> cluster with a nice ERROR message in the logs explaining that multicast
>> wasn't working properly on interface X.
> 
> Can you create a JIRA issue for this ?
> 
https://jira.jboss.org/jira/browse/JGRP-857

-- 
Brian Stansberry
Lead, AS Clustering
JBoss, a division of Red Hat
brian.stansberry@...

gray wrote:
>
> Brian Stansberry-4 wrote:
>> One category of errors JGroups users see is improperly configured
>> multicast on the interface the transport is using.
>>
>> Could the UDP transport help here? Perhaps send a multicast message 
>> during
>> the channel connect phase, and block a short period on a future 
>> waiting to
>> receive your own message. If the message isn't received, log an ERROR
>> with a nice helpful diagnostic.
>>
>
> Yeah, exactly right. We've had tons of issues with multicast routing.
> Anyone setting up a mature networking infrastructure should first, IMO,
> worry about multicast connectivity. For example, Cisco firewall settings
> through our core routers had a couple of multicast obtuse settings that
> we've had to massage. We recently moved data centers hot/hot and had a ton
> of multicast tunneling issues. We've had to log 3-4 bugs with Cisco as
> we've seen complete and partial multicast failures when our configurations
> seemed to be good. We've gotten a bunch of "just reboot it" answers which
> have worked and not helped our risk analysis numbers.

You mean real bugs in the Cisco firmware ? I'm aware of at least one bug 
in the Catalyst firmware (patched in later versions) with IGMP Snooping 
which causes loss of all cached snooping information and subsequent 
dropping of all mcast packets until IGMP snooping has established that 
information again...

> After a particularly bad service failure we decided to add multicast
> connectivity testing to the VIEW_SYNC protocol since it is already
> multicasting around the views. We have a JMX button which sends out a
> multicast test view and expects everyone to ACK it. It can sleep and 
> report
> back on the number of misses.
>
> We also talked [a bit] about some sort of connection testing to stop a 
> node
> from joining the network unless it can prove they have good full-duplex
> multicast. A node can actually join a channel without it as long as the
> coordinator is ok. However, it will immediately get behind on the 
> views and
> can (I think) freeze other nodes from joining since the new views are 
> never
> ACK'd. This really horks the coordinator.

Please comment on https://jira.jboss.org/jira/browse/JGRP-857.

I'm thinking I could (ab)use PING to verify correct working of the 
multicast socket: if we don't receive our own response during discovery, 
we'd log a warning message

-- 
Bela Ban
Lead JGroups / Clustering Team
JBoss - a division of Red Hat

Brian Stansberry wrote:
> I'll have to dig through the jboss-dev list archive to find the exact
> situation referred to in my last paragraph. It was something where
> multicast wasn't working properly on lo on one of Adrian Brock's
> machines, probably no multicast route or something

You don't need a multicast route, as multicasts are in general not 
consulting the (unicast) routing table ! When you define a bind address, 
we use that address to send and receive multicast packets and ignore 
what's in the routing table. Only when there is no bind address, does 
(some) impls consult the routing table, but that behavior is 
implementation dependent.


>> I can do that but the issue is that (a) this will add to the startup 
>> time,
>
> I wouldn't think it would add much though in the standard case where the
> OS passes the multicast packet right back.

Right. This would be the default, so we'd usually not have to wait.


>
> (b) if no response is received, would you want to abort channel
>
> No. Others might have different opinion (e.g. gray's comment on this
> thread where he's looking for something more sophisticated) but all I
> was looking for was a nice ERROR message in the log, so if things go
> wrong, people can see it right at the beginning.


OK. No worries that this message might be overlooked ? ERROR or WARN ?


>
> (c) what if the UDP packet is simply lost (send multiple packets ?).
>
> I'd say no, since in that case all we'd be doing is logging a spurious
> ERROR message. I wouldn't think the odds of not receiving your own
> multicast packet would be high.

OK


>
>> If I do this, we'd have to be able to turn this off, and that would be
>> the default.
>>
>
> No problem. Probably set a timeout for how long to wait with a 
> default  that disables this.

OK.


I'm thinking whether we could reuse the PING (multicast discovery) 
service for this. If we don't receive our own discovery multicast 
packet, we'd flag an error. However, PING can also be configured to 
consult a GossipRouter (unusual config though), so then we wouldn't be 
able to do this.


-- 
Bela Ban
Lead JGroups / Clustering Team
JBoss - a division of Red Hat

You asked me these in our IM chat; quick recap for the list...

Bela Ban wrote:
> 
> 
> Brian Stansberry wrote:
>> I'll have to dig through the jboss-dev list archive to find the exact
>> situation referred to in my last paragraph. It was something where
>> multicast wasn't working properly on lo on one of Adrian Brock's
>> machines, probably no multicast route or something
> 
> You don't need a multicast route, as multicasts are in general not 
> consulting the (unicast) routing table ! When you define a bind address, 
> we use that address to send and receive multicast packets and ignore 
> what's in the routing table. Only when there is no bind address, does 
> (some) impls consult the routing table, but that behavior is 
> implementation dependent.
> 
> 
>>> I can do that but the issue is that (a) this will add to the startup 
>>> time,
>>
>> I wouldn't think it would add much though in the standard case where the
>> OS passes the multicast packet right back.
> 
> Right. This would be the default, so we'd usually not have to wait.
> 
> 
>>
>> (b) if no response is received, would you want to abort channel
>>
>> No. Others might have different opinion (e.g. gray's comment on this
>> thread where he's looking for something more sophisticated) but all I
>> was looking for was a nice ERROR message in the log, so if things go
>> wrong, people can see it right at the beginning.
> 
> 
> OK. No worries that this message might be overlooked ? ERROR or WARN ?
> 

WARN, since with your PING idea below this check can occur with every 
MERGE run. With more frequent checking, chances of a spurious dropped 
packet are higher, so let's make it WARN.

I'm not that concerned about being overlooked. I'm just interested in 
being helpful here; people still have to make some effort. In the 
troublesome cases I saw most, there are problems connecting, so we'll be 
logging a WARN right in the middle of the normal connect logging, i.e. 
where people will be looking.

> 
>>
>> (c) what if the UDP packet is simply lost (send multiple packets ?).
>>
>> I'd say no, since in that case all we'd be doing is logging a spurious
>> ERROR message. I wouldn't think the odds of not receiving your own
>> multicast packet would be high.
> 
> OK
> 
> 
>>
>>> If I do this, we'd have to be able to turn this off, and that would be
>>> the default.
>>>
>>
>> No problem. Probably set a timeout for how long to wait with a 
>> default  that disables this.
> 
> OK.
> 
> 
> I'm thinking whether we could reuse the PING (multicast discovery) 
> service for this. If we don't receive our own discovery multicast 
> packet, we'd flag an error. However, PING can also be configured to 
> consult a GossipRouter (unusual config though), so then we wouldn't be 
> able to do this.
> 
> 

Using PING sounds good!


-- 
Brian Stansberry
Lead, AS Clustering
JBoss, a division of Red Hat
brian.stansberry@...

OK, I'll implement this today

Brian Stansberry wrote:
> You asked me these in our IM chat; quick recap for the list...
>
> Bela Ban wrote:
>>
>>
>> Brian Stansberry wrote:
>>> I'll have to dig through the jboss-dev list archive to find the exact
>>> situation referred to in my last paragraph. It was something where
>>> multicast wasn't working properly on lo on one of Adrian Brock's
>>> machines, probably no multicast route or something
>>
>> You don't need a multicast route, as multicasts are in general not 
>> consulting the (unicast) routing table ! When you define a bind 
>> address, we use that address to send and receive multicast packets 
>> and ignore what's in the routing table. Only when there is no bind 
>> address, does (some) impls consult the routing table, but that 
>> behavior is implementation dependent.
>>
>>
>>>> I can do that but the issue is that (a) this will add to the 
>>>> startup time,
>>>
>>> I wouldn't think it would add much though in the standard case where 
>>> the
>>> OS passes the multicast packet right back.
>>
>> Right. This would be the default, so we'd usually not have to wait.
>>
>>
>>>
>>> (b) if no response is received, would you want to abort channel
>>>
>>> No. Others might have different opinion (e.g. gray's comment on this
>>> thread where he's looking for something more sophisticated) but all I
>>> was looking for was a nice ERROR message in the log, so if things go
>>> wrong, people can see it right at the beginning.
>>
>>
>> OK. No worries that this message might be overlooked ? ERROR or WARN ?
>>
>
> WARN, since with your PING idea below this check can occur with every 
> MERGE run. With more frequent checking, chances of a spurious dropped 
> packet are higher, so let's make it WARN.
>
> I'm not that concerned about being overlooked. I'm just interested in 
> being helpful here; people still have to make some effort. In the 
> troublesome cases I saw most, there are problems connecting, so we'll 
> be logging a WARN right in the middle of the normal connect logging, 
> i.e. where people will be looking.
>
>>
>>>
>>> (c) what if the UDP packet is simply lost (send multiple packets ?).
>>>
>>> I'd say no, since in that case all we'd be doing is logging a spurious
>>> ERROR message. I wouldn't think the odds of not receiving your own
>>> multicast packet would be high.
>>
>> OK
>>
>>
>>>
>>>> If I do this, we'd have to be able to turn this off, and that would be
>>>> the default.
>>>>
>>>
>>> No problem. Probably set a timeout for how long to wait with a 
>>> default  that disables this.
>>
>> OK.
>>
>>
>> I'm thinking whether we could reuse the PING (multicast discovery) 
>> service for this. If we don't receive our own discovery multicast 
>> packet, we'd flag an error. However, PING can also be configured to 
>> consult a GossipRouter (unusual config though), so then we wouldn't 
>> be able to do this.
>>
>>
>
> Using PING sounds good!
>
>

-- 
Bela Ban
Lead JGroups / Clustering Team
JBoss - a division of Red Hat

Perhaps someone has seen this behavior before and has some suggestions.
I'm somewhat at a loss at this point.

We get this message once (at startup) ...

	10:01:18,375 INFO  [STDOUT] 10:01:18,374  WARN GMS:138 -
join(10.191.1.42:8800) sent to 10.191.1.43:8800 timed out (after 83000
ms), retrying

then we get this message every 10 seconds...

	10:01:18,380 INFO  [STDOUT] 10:01:18,379 ERROR
ConnectionTable:529 - failed sending data to 10.191.1.43:8800:
java.net.SocketException: Socket closed


These are 2 Linux boxes running JBoss (or WebLogic, I think it changes
at times) in a cluster containing just them.  They share a common LAN
(they're just inches apart on the same rack!).  We don't get this on
Windows/JBoss setups with the identical protocol stack (just differing
TCPPING.initial_hosts settings) which I've attached.

Beyond this problem, we seem to always be getting the "ConnectionTable
... Socket closed" message and it's killing our confidence in the
systems.  What is wrong with our configurations?

You have some weird timeouts, like

- TCP.max_bundle_timeout is 80 seconds ?
- TCPPING.timeout is 83 seconds ?
- FD.timeout = 90 secs ?
- GMS.join_timeout = 83 seconds ?

What's the point of such high timeouts ?  I would actually suprised if 
this worked at all !
I recommend you use a default configuration (e.g. udp.xml)...


Chermak, Jeslie wrote:
> Perhaps someone has seen this behavior before and has some suggestions.
> I'm somewhat at a loss at this point.
>
> We get this message once (at startup) ...
>
> 10:01:18,375 INFO [STDOUT] 10:01:18,374 WARN GMS:138 -
> join(10.191.1.42:8800) sent to 10.191.1.43:8800 timed out (after 83000
> ms), retrying
>
> then we get this message every 10 seconds...
>
> 10:01:18,380 INFO [STDOUT] 10:01:18,379 ERROR
> ConnectionTable:529 - failed sending data to 10.191.1.43:8800:
> java.net.SocketException: Socket closed
>
>
> These are 2 Linux boxes running JBoss (or WebLogic, I think it changes
> at times) in a cluster containing just them. They share a common LAN
> (they're just inches apart on the same rack!). We don't get this on
> Windows/JBoss setups with the identical protocol stack (just differing
> TCPPING.initial_hosts settings) which I've attached.
>
> Beyond this problem, we seem to always be getting the "ConnectionTable
> ... Socket closed" message and it's killing our confidence in the
> systems. What is wrong with our configurations?
>
> ------------------------------------------------------------------------

-- 
Bela Ban
Lead JGroups / Clustering Team
JBoss - a division of Red Hat

We tried "normal" timeouts to no effect.  The extra 80 seconds were
added to most timeouts to allow for 80 second garbage collections that
our environment sometimes experiences (with 15 GB JVMs).  Adding the
extra time did not affect the result.  We were concerned that not adding
such GC allowance might cause issues, and adding them would allow
sufficient time for all interconnects to work.  We were wrong -- the
extra time does not appear to help the communication between cluster
members.

And we need TCP (not UDP) due to the networks (WANs) we operate in.

-----Original Message-----
From: Bela Ban [mailto:belaban@...] 
Sent: Saturday, December 06, 2008 3:51 AM
To: Chermak, Jeslie
Cc: jGroups-developper List
Subject: Re: [jgroups-dev] unable to JOIN after 83 seconds

You have some weird timeouts, like

- TCP.max_bundle_timeout is 80 seconds ?
- TCPPING.timeout is 83 seconds ?
- FD.timeout = 90 secs ?
- GMS.join_timeout = 83 seconds ?

What's the point of such high timeouts ?  I would actually suprised if 
this worked at all !
I recommend you use a default configuration (e.g. udp.xml)...


Chermak, Jeslie wrote:
> Perhaps someone has seen this behavior before and has some
suggestions.
> I'm somewhat at a loss at this point.
>
> We get this message once (at startup) ...
>
> 10:01:18,375 INFO [STDOUT] 10:01:18,374 WARN GMS:138 -
> join(10.191.1.42:8800) sent to 10.191.1.43:8800 timed out (after 83000
> ms), retrying
>
> then we get this message every 10 seconds...
>
> 10:01:18,380 INFO [STDOUT] 10:01:18,379 ERROR
> ConnectionTable:529 - failed sending data to 10.191.1.43:8800:
> java.net.SocketException: Socket closed
>
>
> These are 2 Linux boxes running JBoss (or WebLogic, I think it changes
> at times) in a cluster containing just them. They share a common LAN
> (they're just inches apart on the same rack!). We don't get this on
> Windows/JBoss setups with the identical protocol stack (just differing
> TCPPING.initial_hosts settings) which I've attached.
>
> Beyond this problem, we seem to always be getting the "ConnectionTable
> ... Socket closed" message and it's killing our confidence in the
> systems. What is wrong with our configurations?
>
>
------------------------------------------------------------------------

-- 
Bela Ban
Lead JGroups / Clustering Team
JBoss - a division of Red Hat

There has been a suggestion that Oracle 10g RAC, running on the same
machine, might be causing the problems.  Currently we get the following
messages at various times:
     WARN JChannel:384 - waiting on UNBLOCK
     MuxChannel:173 - Flush failed
     Multiplexer:657 - failed to collect all service ACKs and the other
server...
     WARN GMS:419 - failed to collect all ACKs
... plus (due, no doubt, to the added 80 second timeouts) a minute-plus
delay after the "GMS: address" message before the startup process
resumes.

We also investigated having "hugepages" on this Linux server as a
culprit and were able to dismiss that possibility.

And we are suspicious of the role loopbacks (outside of the protocol
stack) may be playing in all this.

I am considering restoring the pre-GC timeouts (which lack the 80 second
overhead) to the system.  When a machine goes silent due to a JVM GC, it
will just be marked as "suspect" and go through the normal cycle to
rejoin the cluster.  Since, while in GC mode, it does nothing that the
user will notice, it effectively went to sleep for the duration and now
just needs to reconnect to the cluster.  Opinions?

I've also noticed that there is a gc_* parameter -- what's its function
and should that be used in this application?

-----Original Message-----
From: Bela Ban [mailto:belaban@...] 
Sent: Saturday, December 06, 2008 3:51 AM
To: Chermak, Jeslie
Cc: jGroups-developper List
Subject: Re: [jgroups-dev] unable to JOIN after 83 seconds

You have some weird timeouts, like

- TCP.max_bundle_timeout is 80 seconds ?
- TCPPING.timeout is 83 seconds ?
- FD.timeout = 90 secs ?
- GMS.join_timeout = 83 seconds ?

What's the point of such high timeouts ?  I would actually suprised if 
this worked at all !
I recommend you use a default configuration (e.g. udp.xml)...

Chermak, Jeslie wrote:
> There has been a suggestion that Oracle 10g RAC, running on the same
> machine, might be causing the problems. Currently we get the following
> messages at various times:
> WARN JChannel:384 - waiting on UNBLOCK
> MuxChannel:173 - Flush failed
> Multiplexer:657 - failed to collect all service ACKs and the other
> server...
> WARN GMS:419 - failed to collect all ACKs
> ... plus (due, no doubt, to the added 80 second timeouts) a minute-plus
> delay after the "GMS: address" message before the startup process
> resumes.
>
> We also investigated having "hugepages" on this Linux server as a
> culprit and were able to dismiss that possibility.

There's a wiki describing hugepages on Linux in conjunction with a JBoss 
cluster, forgot where, but google should help. I think it was written by 
Andy Miller for this year's JBossWorld.

> And we are suspicious of the role loopbacks (outside of the protocol
> stack) may be playing in all this.
>
> I am considering restoring the pre-GC timeouts (which lack the 80 second
> overhead) to the system. When a machine goes silent due to a JVM GC, it
> will just be marked as "suspect" and go through the normal cycle to
> rejoin the cluster.

I think you can prevent this by using FD_SOCK, then adding FD_ALL with 
*high* timeouts. This way, crashed nodes will get suspected real quick, 
but crashed switches, kernel panics, or plugged cables will take over 80 
secs to detect. But the latter should be the exception anyway...

> Since, while in GC mode, it does nothing that the
> user will notice, it effectively went to sleep for the duration and now
> just needs to reconnect to the cluster. Opinions?
>
> I've also noticed that there is a gc_* parameter -- what's its function
> and should that be used in this application?

gc_lag ? That was used as a buffer to prevent purging of messages, now 
it is obsolete, I might remove it soon.

-- 
Bela Ban
Lead JGroups / Clustering Team
JBoss - a division of Red Hat

OK, I can easily believe that high timeouts are bad.  But there is
nothing in the documentation that helps to select good timeouts.  I'm
sorry, but just saying "copy one of the known working protocol stacks"
just makes JGroups behave as a black box to those very people who should
understand the stacks.  If there are limits to timeouts, they should be
documented and/or programmatically enforced.

15GB JVMs are not by choice: we are working with millions of objects
within a partitioned database.  Things will get worse before they get
better.  And yes, we've already done GC tuning: the 80 second GC pause
occurs about once a day, a handful of 20 second pauses occur daily, and
the remaining GC pauses are less than half a second.  We run 24/7 and
when a particular installation requires monster memory we will
eventually be subject to a massive GC pause.

Because the ACK and other messages are not understood, and because there
is not enough information to help us correct these messages, there will
always be the underlying fear that something is broken in some
unspecified way.  Of course we'll figure it out when it eventually
breaks non-benignly at a customer site (we'll finally see the
consequence of the ignored messages), but that is the very fear which
makes us suspect the correctness of the JGroups protocol stack to begin
with.  Unless we can understand the root causes (and cures) of these
messages (including the warnings), our use of JGroups is crippled.

Please make a wizard to help with this.  I know it's on the to-do list
for JGroups.

Please fully document all protocol stack properties and their effect.
There has been a vast improvement within the past month, but there is
both outdated information for properties that have been removed and
incomplete/no information regarding properties that exist.

Chermak, Jeslie wrote:
> OK, I can easily believe that high timeouts are bad. But there is
> nothing in the documentation that helps to select good timeouts.

Timeouts are very application-dependent. In 
http://www.jboss.org/community/docs/DOC-9739, for example, the merits 
(and disadvantages) of timeouts versus socket connections for failure 
detection is discussed.

> I'm sorry, but just saying "copy one of the known working protocol stacks"
> just makes JGroups behave as a black box to those very people who should
> understand the stacks. If there are limits to timeouts, they should be
> documented and/or programmatically enforced.

http://www.jgroups.org/javagroupsnew/docs/manual/html/protlist.html#d0e4110 
discusses timeouts for failure detection.

> 15GB JVMs are not by choice: we are working with millions of objects
> within a partitioned database. Things will get worse before they get
> better. And yes, we've already done GC tuning: the 80 second GC pause
> occurs about once a day, a handful of 20 second pauses occur daily, and
> the remaining GC pauses are less than half a second. We run 24/7 and
> when a particular installation requires monster memory we will
> eventually be subject to a massive GC pause.
>
> Because the ACK and other messages are not understood, and because there
> is not enough information to help us correct these messages, there will
> always be the underlying fear that something is broken in some
> unspecified way. Of course we'll figure it out when it eventually
> breaks non-benignly at a customer site (we'll finally see the
> consequence of the ignored messages), but that is the very fear which
> makes us suspect the correctness of the JGroups protocol stack to begin
> with. Unless we can understand the root causes (and cures) of these
> messages (including the warnings), our use of JGroups is crippled.
>
> Please make a wizard to help with this. I know it's on the to-do list
> for JGroups.

Yes, it is on the todo list, but not with a high priority. There's only 
a limited set of resources working on JGroups, and we have to tackle 
tasks in order of priority.


-- 
Bela Ban
Lead JGroups / Clustering Team
JBoss - a division of Red Hat

Chermak, Jeslie wrote:
> We tried "normal" timeouts to no effect. The extra 80 seconds were
> added to most timeouts to allow for 80 second garbage collections that
> our environment sometimes experiences (with 15 GB JVMs).

First of all, I can *assure* you that those high timeouts will lead to 
unexpected behavior, if not outright failures, in JGroups !

Second, I think JVMs with 15GB heap size are bad, I don't think that 
JVMs have really obeen ptimized for those large heap sizes just yet. 
Unless your app really demands such a large heap size, I would rather 
split it into multiple smaller JVM processes.

> Adding the
> extra time did not affect the result. We were concerned that not adding
> such GC allowance might cause issues, and adding them would allow
> sufficient time for all interconnects to work. We were wrong -- the
> extra time does not appear to help the communication between cluster
> members.

I'm sure you can tune the garbage collector such that it doesn't 
stop-the-world for 80 seconds, but instead GC's in the background, 
continually.

-- 
Bela Ban
Lead JGroups / Clustering Team
JBoss - a division of Red Hat

Chermak, Jeslie wrote:
> Thanks for the references. Unfortunately I'd already found and read
> them. They do not provide enough information.

 From a users point of view, they do (IMO).

If you're interested in the implementation, read the code. The comments 
inside the code are also helpful, e.g. for FD_ALL:
/**
 * Failure detection based on simple heartbeat protocol. Every member
 * periodically multicasts a heartbeat. Every member also maintains a 
table of
 * all members (minus itself). When data or a heartbeat from P are 
received, we
 * reset the timestamp for P to the current time. Periodically, we check for
 * expired members, and suspect those.
 *
 * @author Bela Ban
 * @version $Id: FD_ALL.java,v 1.25 2008/10/22 10:29:15 belaban Exp $
 */

> For example, what really
> distinguishes FD (which we use) from FD_ALL (which you recommended).
> And there is no mention of FD_ALL in the FD vs. FD_SOCK article.

What applies to FD applies to FD_ALL, too (with respect to timeouts).

-- 
Bela Ban
Lead JGroups / Clustering Team
JBoss - a division of Red Hat

Let's continue the discussion on jg-dev (copied), so we have a 
searchable archive of Qs and As and other folks can find help there...

Chermak, Jeslie wrote:
> For some reason I thought the GC overhead needed to be added to all
> levels. Now that I think about it, it makes more sense to only allow it
> in the failure detection logic.
>
> As to FD versus FD_ALL, the documentation implies that both are
> heartbeat-related protocols, so it is not clear which to select under
> what circumstances.

FD: use with either TCP or UDP. FD_ALL: use with UDP only.

Besides that: with FD, a suspicion message is triggered by only *one* 
member, whereas FD_ALL requires consensus between different members to 
trigger a SUSPECT.

> BTW, thanks for taking the time to answer my e-mails. I really do 
> appreciate it.

-- 
Bela Ban
Lead JGroups / Clustering Team
JBoss - a division of Red Hat