#1542 Interruption flag is cleared during statement execution

[raphw]

I encountered an issue with an application that was occasionally leaking active threads from thread pools that were shutdown. I traced this problem back to HSQLDB suppressing interruption flags during its statement execution. I saw this to be a pattern in for example the Session class where Thread.interrupted() is invoked after catching an interruption exception what clears the flag.

As a consequence, a thread that is currently executing a statement will not be able to shut down as the interruption signal is only sent once. If the thread's event loop is checking for the flag to be set, it will have been cleared by HSQLDB. This is breaking the contract of thread interrupts making ordered shutdown impossible.

Is there a particular reason for you to clear that flag? If it should not be set during the session execution, it would be important to self-interrupt the thread before returing to invoking code.

[Fred Toussi]

During execution of statements, HSQLDB controls the running threads. It is designed this way. You should SHUTDOWN the in-process database before shutting down the thread pool.

[raphw]

This is not feasible in many cases. Take this scenario for example: A user controls multiple background task via a GUI and wants to start and stop individual tasks. To stop a task, the specifc thread is sent an interrupt to signal its termination. If that thread is by chance executing a JDBC statement in the particular moment, this wish for termination will be suppressed. The only workaround is currently to spam the thread with interrupts to hopefully hit a moment when no JDBC statement is executed. If querying the database is a significant part of the thread's work, this requires a high frequency. Shutting down the datasource would in contrast require to end all tasks what is not desired behavior.

If HSQLDB does not propagate the interrupt signal, code using it will malfunction. Interruption is a thread-local state that needs to be visible to any event-loop that is typically implementing a runnable's run method.

[Fred Toussi]

I've seen the photo on Twitter. Hi Rafael.

Within a program, an interrupt is often used for termination, but it can be used for other purposes. For example a Java turotial states "An interrupt is an indication to a thread that it should stop what it is doing and do something else. It's up to the programmer to decide exactly how a thread responds to an interrupt, but it is very common for the thread to terminate." (https://docs.oracle.com/javase/tutorial/essential/concurrency/interrupt.html)

But some programs or libraries may use an interrupt at a higher level without realizing what the thread is actually doing and cause malfunction--and I have had such behaviour reported. I do understand your requirement that only one task/thread may need termination without affecting other tasks accessing the database. So I will add a user setting to allow the session to respond to the interrupt and rollback the operation and return an exception without closing the connection or the database.

The current dev code is in the SVN /branches/dev-three branch and will be updated soon with this fix.

The code has been recently reviewed to remove all usage of deprecated methods. Please have a look at this usage of Thread::stop() https://sourceforge.net/p/hsqldb/svn/HEAD/tree/base/branches/dev-three/src/org/hsqldb/jdbc/JDBCDriver.java which returns a connection, with a limit on the time it takes to establish connection. Let me know if you can suggest a better alternative strategy to implement this, perhaps using more recent Java libraries.

[raphw]

Hi Fred,

thanks for your detailed answer, I am glad that you are adressing the problem. I am using HSQLDB in a unit test scenario. We do also test everything in Docker against Oracle and Postgres databases that we use in production but for quick feedback HSQLDB works amazingly well as a substitute for some tests, even for the more complicated queries. Congrationaltions to you for maintaing such a magnificent tool!

As for the JDBC behavior, the swallowed interrupts are a major difference between using Oracle/Postgres and HSQLDB. I assume just for this use case it makes perfect sense to not clearing the flag, so thanks for taking this in.

Now this is my understanding of thread interruption, I might be wrong and I plan to research it more, the interruption state is considered to be a form of thread-local state that needs to be kept consistent by all code that interacts with that part of the flag. Many concurrency frameworks run a dispatcher similar to this:

    Thread thread = new Thread(() -> {
        try {
            while (!Thread.interrupted()) {
                runSomeUserCode(); // throws InterruptedException
                Thread.sleep(1000); // pause
            }
        } catch (InterruptedException shutdown) { }
    });
    thread.start();
    Thread.sleep(100);
    thread.interrupt(); // tells event loop to complete

In our case where I encountered the issue with using HSQLDB, this is certainly the case. In this case, interrupting a thread is considered a standard API of telling a thread to stop its work without the thread's user needing to know any detail of the thread's implementation.

As you stated, Oracle describes interruption being used to shut down a thread in most cases. Since JDBC is agnostic to concurrency and does not document any different behavior, I would expect the flag to remain untouched when calling a method. As a matter of fact, I consider it an oversight that the JDBC methods do not throw interrupt exceptions; after all they typically represent a socket connection and I have before worked with monitoring threads that call Statement.cancel for a long running queryin another thread that was interrupted during execution which seems to be the only workaround to this.

I would therefore suggest that you implement interrupting code in HSQLDB similar to:

    try {
        doSomethingInHsqldb();
    } catch (InterruptedException ie) {
        Thread.currentThread().interrupt();
    }

If you do not require interruptions at all anymore, even better. In any case, the interruption state would be correctly propagated to the user allowing threads to shut down correctly.

Thanks for your amazing work and adressing this issue!
Rafael

[Fred Toussi]

Fixed and committed to trunk. On interrupt, the current statement is completed, the transaction is rolled back and interrupt flag is set again. Could you check with your setup.

[raphw]

I'd recommend to not rollback the transaction. I would not expect this to have happened from an interrupt.

[Fred Toussi]

The interrupt was cleared only while the session thread was waiting to execute a statement that was previously submitted. This is because the latch would throw an exception with the flag set. We now record the fact the interrupt occured and when the latch is no longer in force, we roll back.

If you are saying we should change it so that the session continues after the interrupt, the session cannot work properly with the interrupt flag set. The next execution will probably hit the same latch.

I may add a user setting to choose between the old and the new behaviour.

[raphw]

I see, but I am wondering: is there an exception thrown? How would I know as a user that the statement was rolled back? The interrupt could have been set just after the statement completed; is there an exception thrown?

[Fred Toussi]

Yes, an SQLException is thrown when the transaction has been rolled back.

The InterruptedException is thrown only while waiting to execute. It the user app sets the interrupt during execution and the transaction does not have to wait, the statement or transaction is not rolled back.

[raphw]

Great, then this will work just fine for the use cases that I had in mind! Thanks for taking care of this.