activate failed with jgroups, StreamCorruptedException

[Yee_Keat Phuah]

I am using ha-jdbc-3.0.4-snapshot, latest from github. With JGroups 3.6.3.

Both the first and second app starts up fine, but when I trigger a "activate" command on one of the app, I get the following exception:
2015-04-02 11:09:39,758 WARN net.sf.hajdbc.lock.distributed.DistributedLockManager [?:?] Failed to acquire writeLock() on gateway-dr-243
java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
at net.sf.hajdbc.distributed.jgroups.JGroupsCommandDispatcher$RspCommandResponse.get(JGroupsCommandDispatcher.java:347) ~[ha-jdbc-3.0.4-SNAPSHOT.jar:3.0.4-SNAPSHOT]
at net.sf.hajdbc.lock.distributed.DistributedLockManager$DistributedLock.readAcquireResponse(DistributedLockManager.java:524) [ha-jdbc-3.0.4-SNAPSHOT.jar:3.0.4-SNAPSHOT]
at net.sf.hajdbc.lock.distributed.DistributedLockManager$DistributedLock.lockMembers(DistributedLockManager.java:481) [ha-jdbc-3.0.4-SNAPSHOT.jar:3.0.4-SNAPSHOT]
at net.sf.hajdbc.lock.distributed.DistributedLockManager$DistributedLock.lockInterruptibly(DistributedLockManager.java:330) [ha-jdbc-3.0.4-SNAPSHOT.jar:3.0.4-SNAPSHOT]
at net.sf.hajdbc.sql.DatabaseClusterImpl.activate(DatabaseClusterImpl.java:864) [ha-jdbc-3.0.4-SNAPSHOT.jar:3.0.4-SNAPSHOT]
at net.sf.hajdbc.sql.DatabaseClusterImpl.activate(DatabaseClusterImpl.java:156) [ha-jdbc-3.0.4-SNAPSHOT.jar:3.0.4-SNAPSHOT]
at net.sf.hajdbc.sql.DatabaseClusterImpl.activate(DatabaseClusterImpl.java:134) [ha-jdbc-3.0.4-SNAPSHOT.jar:3.0.4-SNAPSHOT]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_45]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_45]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_45]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_45]
at net.sf.hajdbc.management.AnnotatedMBean.invoke(AnnotatedMBean.java:214) [ha-jdbc-3.0.4-SNAPSHOT.jar:3.0.4-SNAPSHOT]
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819) [na:1.7.0_45]
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801) [na:1.7.0_45]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_45]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_45]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_45]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_45]
at com.google.web.bindery.requestfactory.server.ReflectiveServiceLayer.invoke(ReflectiveServiceLayer.java:182) [gwt-servlet.jar:na]
at com.google.web.bindery.requestfactory.server.ServiceLayerDecorator.invoke(ServiceLayerDecorator.java:111) [gwt-servlet.jar:na]
at com.google.web.bindery.requestfactory.server.ServiceLayerDecorator.invoke(ServiceLayerDecorator.java:111) [gwt-servlet.jar:na]
at com.google.web.bindery.requestfactory.server.SimpleRequestProcessor.processInvocationMessages(SimpleRequestProcessor.java:463) [gwt-servlet.jar:na]
at com.google.web.bindery.requestfactory.server.SimpleRequestProcessor.process(SimpleRequestProcessor.java:233) [gwt-servlet.jar:na]
at com.google.web.bindery.requestfactory.server.SimpleRequestProcessor.process(SimpleRequestProcessor.java:135) [gwt-servlet.jar:na]
at com.google.web.bindery.requestfactory.server.RequestFactoryServlet.doPost(RequestFactoryServlet.java:133) [gwt-servlet.jar:na]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:647) [tomcat-servlet-3.0-api.jar:na]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:728) [tomcat-servlet-3.0-api.jar:na]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) [catalina.jar:7.0.33]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) [catalina.jar:7.0.33]
at org.springframework.orm.jpa.support.OpenEntityManagerInViewFilter.doFilterInternal(OpenEntityManagerInViewFilter.java:177) [spring-orm-4.0.0.RELEASE.jar:4.0.0.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:108) [spring-web-4.0.0.RELEASE.jar:4.0.0.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) [catalina.jar:7.0.33]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) [catalina.jar:7.0.33]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222) [catalina.jar:7.0.33]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) [catalina.jar:7.0.33]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472) [catalina.jar:7.0.33]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) [catalina.jar:7.0.33]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99) [catalina.jar:7.0.33]
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:931) [catalina.jar:7.0.33]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) [catalina.jar:7.0.33]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407) [catalina.jar:7.0.33]
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004) [tomcat-coyote.jar:7.0.33]
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589) [tomcat-coyote.jar:7.0.33]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1653) [tomcat-coyote.jar:7.0.33]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_45]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
Caused by: java.lang.reflect.InvocationTargetException: null
at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:476) ~[jgroups-3.6.3.Final.jar:3.6.3.Final]
at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:377) ~[jgroups-3.6.3.Final.jar:3.6.3.Final]
at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:250) ~[jgroups-3.6.3.Final.jar:3.6.3.Final]
at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:675) ~[jgroups-3.6.3.Final.jar:3.6.3.Final]
at org.jgroups.JChannel.up(JChannel.java:739) ~[jgroups-3.6.3.Final.jar:3.6.3.Final]
at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1029) ~[jgroups-3.6.3.Final.jar:3.6.3.Final]
at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:488) ~[jgroups-3.6.3.Final.jar:3.6.3.Final]
at org.jgroups.protocols.pbcast.STATE_TRANSFER.up(STATE_TRANSFER.java:146) ~[jgroups-3.6.3.Final.jar:3.6.3.Final]
at org.jgroups.protocols.RSVP.up(RSVP.java:201) ~[jgroups-3.6.3.Final.jar:3.6.3.Final]
at org.jgroups.protocols.FRAG2.up(FRAG2.java:165) ~[jgroups-3.6.3.Final.jar:3.6.3.Final]
at org.jgroups.protocols.FlowControl.up(FlowControl.java:383) ~[jgroups-3.6.3.Final.jar:3.6.3.Final]
at org.jgroups.protocols.FlowControl.up(FlowControl.java:394) ~[jgroups-3.6.3.Final.jar:3.6.3.Final]
at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1045) ~[jgroups-3.6.3.Final.jar:3.6.3.Final]
at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234) ~[jgroups-3.6.3.Final.jar:3.6.3.Final]
at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:435) ~[jgroups-3.6.3.Final.jar:3.6.3.Final]
at org.jgroups.protocols.pbcast.NAKACK2.deliver(NAKACK2.java:961) ~[jgroups-3.6.3.Final.jar:3.6.3.Final]
at org.jgroups.protocols.pbcast.NAKACK2.handleMessage(NAKACK2.java:843) ~[jgroups-3.6.3.Final.jar:3.6.3.Final]
at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:618) ~[jgroups-3.6.3.Final.jar:3.6.3.Final]
at org.jgroups.protocols.BARRIER.up(BARRIER.java:152) ~[jgroups-3.6.3.Final.jar:3.6.3.Final]
at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155) ~[jgroups-3.6.3.Final.jar:3.6.3.Final]
at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:200) ~[jgroups-3.6.3.Final.jar:3.6.3.Final]
at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:299) ~[jgroups-3.6.3.Final.jar:3.6.3.Final]
at org.jgroups.protocols.MERGE2.up(MERGE2.java:237) ~[jgroups-3.6.3.Final.jar:3.6.3.Final]
at org.jgroups.protocols.Discovery.up(Discovery.java:291) ~[jgroups-3.6.3.Final.jar:3.6.3.Final]
at org.jgroups.protocols.TP$ProtocolAdapter.up(TP.java:2842) ~[jgroups-3.6.3.Final.jar:3.6.3.Final]
at org.jgroups.protocols.TP.passMessageUp(TP.java:1577) ~[jgroups-3.6.3.Final.jar:3.6.3.Final]
at org.jgroups.protocols.TP$MyHandler.run(TP.java:1796) ~[jgroups-3.6.3.Final.jar:3.6.3.Final]
... 3 common frames omitted
Caused by: java.lang.IllegalStateException: java.io.StreamCorruptedException: invalid stream header: 19830206
at net.sf.hajdbc.util.Objects.deserialize(Objects.java:153) ~[ha-jdbc-3.0.4-SNAPSHOT.jar:3.0.4-SNAPSHOT]
at net.sf.hajdbc.util.Objects.deserialize(Objects.java:126) ~[ha-jdbc-3.0.4-SNAPSHOT.jar:3.0.4-SNAPSHOT]
at net.sf.hajdbc.distributed.jgroups.JGroupsCommandDispatcher.handle(JGroupsCommandDispatcher.java:235) ~[ha-jdbc-3.0.4-SNAPSHOT.jar:3.0.4-SNAPSHOT]
at org.jgroups.blocks.MessageDispatcher.handle(MessageDispatcher.java:487) ~[jgroups-3.6.3.Final.jar:3.6.3.Final]
at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:472) ~[jgroups-3.6.3.Final.jar:3.6.3.Final]
... 29 common frames omitted
Caused by: java.io.StreamCorruptedException: invalid stream header: 19830206
at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:804) ~[na:1.7.0_45]
at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299) ~[na:1.7.0_45]
at net.sf.hajdbc.util.Objects.deserialize(Objects.java:141) ~[ha-jdbc-3.0.4-SNAPSHOT.jar:3.0.4-SNAPSHOT]
... 33 common frames omitted

[Yee_Keat Phuah]

After some digging, seems to be a bug, should be using the getBuffer() from Message instead of getRawBuffer(), here's the patch:

diff --git a/src/main/java/net/sf/hajdbc/distributed/jgroups/JGroupsCommandDispatcher.java b/src/main/java/net/sf/hajdbc/distributed/jgroups/JGroupsCommandDispatcher.java
index 4775e36..52acf00 100644
---  a/src/main/java/net/sf/hajdbc/distributed/jgroups/JGroupsCommandDispatcher.java
+++  b/src/main/java/net/sf/hajdbc/distributed/jgroups/JGroupsCommandDispatcher.java
 @@ -232,7 +232,7 @@ public class JGroupsCommandDispatcher<C> implements RequestHandler, CommandDispa
    @Override
    public Object handle(Message message)
    {
-       Command<Object, C> command = Objects.deserialize(message.getRawBuffer());
+       Command<Object, C> command = Objects.deserialize(message.getBuffer());

    this.logger.log(Level.DEBUG, Messages.COMMAND_RECEIVED.getMessage(command, message.getSrc()));

[Justin Cranford]

Is the problem related to using JGroups 3.6? The dependencies page says to use JGroups 3.4.3.

http://ha-jdbc.github.io/dependencies.html

I last tried HA-JDBC 3.0.4-SNAPSHOT with JGroups 3.4.6, but I see it is up to 3.4.8 now. Those are just bug fixes, so probably safe to upgrade.

I am wondering if anything major changed between JGroups 3.4 and 3.6 that would cause incompatibility of HA-JDBC 3.0.4-SNAPSHOT usage of JGroups APIs.

[Paul Ferraro]

HA-JDBC 3.0 and JGroups 3.6 are not compatible. You can either recompile with the above source change or stick with JGroups 3.4.

[Yee_Keat Phuah]

The reason why I tried jgroups 3.6 is because I couldn't get the jgroups 3.4 to work (with 3.0.3 or 3.0.4-snapshot), I get this error

2015-04-02 23:31:07,469 ERROR org.jgroups.protocols.UDP [TP.java:1349] JGRP000029: null: failed sending message to cluster (98 bytes): java.lang.Exception: dest=/228.8.8.8:45588 (101 bytes), headers: PING: [PING: type=GET_MBRS_REQ, cluster=gateway.state, view_id=[gateway-29834|0]], UDP: [channel_name=gateway.state]

If I swap in the 3.6 jgroups, then I am able to have the cluster working, with exactly the same configuration.

Is there any other incompatibility that you are aware of with jgroups 3.6?

[Yee_Keat Phuah]

Just an update, the error that I was facing with jgroups 3.4, its because I use System.setProperty(java.net.preferIPv4Stack), once I define it in CATALINA_OPTS for Tomcat then the cluster seemed to be connected properly.