I am using HSQLDB for running integration tests in my project. I am currently migrating from XAPool and JOTM to BTM http://btm.codehaus.org/
While trying out the integration I've noticed that HSQLDB fails to perform rollback when it's connection is enlisted to an XA transaction moderated by BTM. The difference of behavior from JOTM is that it's XAResourceManager associated with transaction implementation object delists all enlisted resources (calls XAResource.end(Xid, int) on them) before a commit or rollback operation is performed. Referring to JTA 1.1 specification this seems to be legitimate behavior, see chapters 3.4.6 and 4.2. Moreover, PostgreSQL 9.1 and Apache Derby 10.9.1.0 cope with this control flow without problems.
HSQLDB JDBCXAResource expects that rollback() will occur only when connection is in state XA_STATE_PREPARED. This is not correct on two accounts - one is that rollback may occur when connection is not associated with a transactoin (XA_STATE_ENDED), second is that transaction manager implementation is allowed to perform "single resource manager optimization" and perform single phase commit, skipping prepare stage. In the latter case the connection never enters XA_STATE_PREPARED thus making rollback impossible under HSQLDB implementation. To resolve this problem I've changed the condition in rollbackThis() method to proceed when with rollback when connection is in XA_STATE_ENDED.
After fixing the above problem I've run into another. more insidious one. UserTransaction.rollback() completes without throwing an exception, but in fact it performs a commit operation! This happens because end(Xid, int) invoked by BTM before rollback(Xid) attempts to restore original autoCommit status of the connection. AutoCommit is off for XA connections, but it's on for regular connections. Changing autocommit flag to a different value causes an automatic commit. I fixed this issue by moving restoring the autoCommit flag to dispose() method that is invoked from both commit(Xid) and rollback(Xid) which preserves the expected autoCommit flag behavior but fixes the inadvertant commit on rollback problem.
I've encountered these problems with HSQLDB 2.2.9, and I've found that the relevant source file was not changed between release and current trunk.
Please find the attached patch against https://hsqldb.svn.sourceforge.net/svnroot/hsqldb/base/trunk revision 5099