[jgroups-users] Hijacking the JGroups channel inside Infinispan and not getting away with it :)

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

Hi all,

We have run into an interesting race condition when attempting to use a
fork channel in our application, we more or less follow what Bela wrote
here
http://belaban.blogspot.com/2013/08/how-to-hijack-jgroups-channel-inside.html
.

When the cluster comes up and receives views the node broadcasts
information about itself over the fork channel so the cluster knows what
each node is capable of handling.

Unfortunately from what I can see there is a race condition on bootstrap
where the JGroups stack is started and receives a fork message before the
fork is inserted into the stack (fork not present in stack trace) [1] which
results in garbage / unknown data passing through the Infinispan
marshaller.. if you are lucky enough it will read an extremely large int
and try and allocate that into a byte array resulting in the JVM to throw
an OOM or NegativeArraySizeException

I believe one possible solution is to define the fork inside the
jgroups.xml which is used to create the initial jgroups stack which would
hopefully discard fork channel messages (until the message listener is
registered) and not pass them up the stack resulting in undefined behaviour.

I had a look at the fork documentation but there are not many examples,
does my possible solution seem feasible or does someone have alternative
solutions? I am currently looking through the Infinispan code to see if
there is any way to decorate jgroups before it starts.

Thanks in advance,

Johnathan

[1]

2022-10-20 10:42:28,515 ERROR [jgroups-89,service-2]
(org.infinispan.CLUSTER) ISPN000474: Error processing request 0@service-2
java.lang.NegativeArraySizeException: -436207616
at
org.infinispan.marshall.core.GlobalMarshaller.readUnknown(GlobalMarshaller.java:904)
~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]
at
org.infinispan.marshall.core.GlobalMarshaller.readUnknown(GlobalMarshaller.java:891)
~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]
at
org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:715)
~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]
at
org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:358)
~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]
at
org.infinispan.marshall.core.GlobalMarshaller.objectFromObjectInput(GlobalMarshaller.java:192)
~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]
at
org.infinispan.marshall.core.GlobalMarshaller.objectFromByteBuffer(GlobalMarshaller.java:221)
~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.processRequest(JGroupsTransport.java:1361)
~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1301)
~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$300(JGroupsTransport.java:130)
~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.lambda$up$0(JGroupsTransport.java:1450)
~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]
at org.jgroups.util.MessageBatch.forEach(MessageBatch.java:318)
~[jgroups-4.2.1.Final.jar:4.2.1.Final]
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.up(JGroupsTransport.java:1450)
~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]
at org.jgroups.JChannel.up(JChannel.java:796)
~[jgroups-4.2.1.Final.jar:4.2.1.Final]
at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:903)
~[jgroups-4.2.1.Final.jar:4.2.1.Final]
at org.jgroups.protocols.FRAG3.up(FRAG3.java:187)
~[jgroups-4.2.1.Final.jar:4.2.1.Final]
at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
~[jgroups-4.2.1.Final.jar:4.2.1.Final]
at org.jgroups.stack.Protocol.up(Protocol.java:338)
~[jgroups-4.2.1.Final.jar:4.2.1.Final]
at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:297)
~[jgroups-4.2.1.Final.jar:4.2.1.Final]
at org.jgroups.protocols.UNICAST3.deliverBatch(UNICAST3.java:1071)
~[jgroups-4.2.1.Final.jar:4.2.1.Final]
at org.jgroups.protocols.UNICAST3.removeAndDeliver(UNICAST3.java:886)
~[jgroups-4.2.1.Final.jar:4.2.1.Final]
at org.jgroups.protocols.UNICAST3.handleBatchReceived(UNICAST3.java:852)
~[jgroups-4.2.1.Final.jar:4.2.1.Final]
at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:501)
~[jgroups-4.2.1.Final.jar:4.2.1.Final]
at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:689)
~[jgroups-4.2.1.Final.jar:4.2.1.Final]
at org.jgroups.stack.Protocol.up(Protocol.java:338)
~[jgroups-4.2.1.Final.jar:4.2.1.Final]
at org.jgroups.protocols.FailureDetection.up(FailureDetection.java:197)
~[jgroups-4.2.1.Final.jar:4.2.1.Final]
at org.jgroups.stack.Protocol.up(Protocol.java:338)
~[jgroups-4.2.1.Final.jar:4.2.1.Final]
at org.jgroups.stack.Protocol.up(Protocol.java:338)
~[jgroups-4.2.1.Final.jar:4.2.1.Final]
at org.jgroups.stack.Protocol.up(Protocol.java:338)
~[jgroups-4.2.1.Final.jar:4.2.1.Final]
at org.jgroups.protocols.TP.passBatchUp(TP.java:1408)
~[jgroups-4.2.1.Final.jar:4.2.1.Final]
at
org.jgroups.util.MaxOneThreadPerSender$BatchHandlerLoop.passBatchUp(MaxOneThreadPerSender.java:284)
~[jgroups-4.2.1.Final.jar:4.2.1.Final]
at
org.jgroups.util.SubmitToThreadPool$BatchHandler.run(SubmitToThreadPool.java:136)
~[jgroups-4.2.1.Final.jar:4.2.1.Final]
at
org.jgroups.util.MaxOneThreadPerSender$BatchHandlerLoop.run(MaxOneThreadPerSender.java:273)
~[jgroups-4.2.1.Final.jar:4.2.1.Final]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
~[?:?]
at java.lang.Thread.run(Thread.java:829) ~[?:?]