Re: [jgroups-users] Hijacking the JGroups channel inside Infinispan and not getting away with it :)

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

Hi Jonathan

yes, the best solution is to define FORK in the JGroups section of the 
Infinispan configuration, as you mentioned.

If you don't control the configuration, then it gets a bit more 
tricky... you could (*before sending any brodcast traffic*) insert FORK 
dynamically into every JChannel instance.
To do that, you need to get the JChannel; IIRC, via (paraphrased) 
cache.getExtendedCache().getRpcManager().getTransport(), downcast it to 
JGroupsTransport, then call getChannel().

Once you have JChannel, call 
channel.getProtocolStack().insertProtocolAtTop(new FORK(...));
Hope this helps,

[1] http://www.jgroups.org/manual5/index.html#ForkChannel

On 21.10.22 16:02, Questions/problems related to using JGroups wrote:
> Hi all,
> 
> We have run into an interesting race condition when attempting to use a 
> fork channel in our application, we more or less follow what Bela wrote 
> here 
> http://belaban.blogspot.com/2013/08/how-to-hijack-jgroups-channel-inside.html <http://belaban.blogspot.com/2013/08/how-to-hijack-jgroups-channel-inside.html> .
> 
> When the cluster comes up and receives views the node broadcasts 
> information about itself over the fork channel so the cluster knows what 
> each node is capable of handling.
> 
> Unfortunately from what I can see there is a race condition on bootstrap 
> where the JGroups stack is started and receives a fork message before 
> the fork is inserted into the stack (fork not present in stack trace) 
> [1] which results in garbage / unknown data passing through the 
> Infinispan marshaller.. if you are lucky enough it will read an 
> extremely large int and try and allocate that into a byte array 
> resulting in the JVM to throw an OOM or NegativeArraySizeException
> 
> I believe one possible solution is to define the fork inside the 
> jgroups.xml which is used to create the initial jgroups stack which 
> would hopefully discard fork channel messages (until the message 
> listener is registered) and not pass them up the stack resulting in 
> undefined behaviour.
> 
> I had a look at the fork documentation but there are not many examples, 
> does my possible solution seem feasible or does someone have alternative 
> solutions? I am currently looking through the Infinispan code to see if 
> there is any way to decorate jgroups before it starts.
> 
> Thanks in advance,
> 
> Johnathan
> 
> [1]
> 
> 2022-10-20 10:42:28,515 ERROR [jgroups-89,service-2] 
> (org.infinispan.CLUSTER) ISPN000474: Error processing request 0@service-2
> java.lang.NegativeArraySizeException: -436207616
> at 
> org.infinispan.marshall.core.GlobalMarshaller.readUnknown(GlobalMarshaller.java:904) ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]
> at 
> org.infinispan.marshall.core.GlobalMarshaller.readUnknown(GlobalMarshaller.java:891) ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]
> at 
> org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:715) ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]
> at 
> org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:358) ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]
> at 
> org.infinispan.marshall.core.GlobalMarshaller.objectFromObjectInput(GlobalMarshaller.java:192) ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]
> at 
> org.infinispan.marshall.core.GlobalMarshaller.objectFromByteBuffer(GlobalMarshaller.java:221) ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]
> at 
> org.infinispan.remoting.transport.jgroups.JGroupsTransport.processRequest(JGroupsTransport.java:1361) ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]
> at 
> org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1301) ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]
> at 
> org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$300(JGroupsTransport.java:130) ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]
> at 
> org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.lambda$up$0(JGroupsTransport.java:1450) ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]
> at org.jgroups.util.MessageBatch.forEach(MessageBatch.java:318) 
> ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> at 
> org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.up(JGroupsTransport.java:1450) ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]
> at org.jgroups.JChannel.up(JChannel.java:796) 
> ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:903) 
> ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> at org.jgroups.protocols.FRAG3.up(FRAG3.java:187) 
> ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) 
> ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> at org.jgroups.stack.Protocol.up(Protocol.java:338) 
> ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:297) 
> ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> at org.jgroups.protocols.UNICAST3.deliverBatch(UNICAST3.java:1071) 
> ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> at org.jgroups.protocols.UNICAST3.removeAndDeliver(UNICAST3.java:886) 
> ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> at org.jgroups.protocols.UNICAST3.handleBatchReceived(UNICAST3.java:852) 
> ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:501) 
> ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:689) 
> ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> at org.jgroups.stack.Protocol.up(Protocol.java:338) 
> ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> at org.jgroups.protocols.FailureDetection.up(FailureDetection.java:197) 
> ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> at org.jgroups.stack.Protocol.up(Protocol.java:338) 
> ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> at org.jgroups.stack.Protocol.up(Protocol.java:338) 
> ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> at org.jgroups.stack.Protocol.up(Protocol.java:338) 
> ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> at org.jgroups.protocols.TP.passBatchUp(TP.java:1408) 
> ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> at 
> org.jgroups.util.MaxOneThreadPerSender$BatchHandlerLoop.passBatchUp(MaxOneThreadPerSender.java:284) ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> at 
> org.jgroups.util.SubmitToThreadPool$BatchHandler.run(SubmitToThreadPool.java:136) ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> at 
> org.jgroups.util.MaxOneThreadPerSender$BatchHandlerLoop.run(MaxOneThreadPerSender.java:273) ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
> at java.lang.Thread.run(Thread.java:829) ~[?:?]
> 
> 
> _______________________________________________
> javagroups-users mailing list
> jav...@li...
> https://lists.sourceforge.net/lists/listinfo/javagroups-users

-- 
Bela Ban | http://www.jgroups.org