Re: [xSocket-develop] Question: SSL with non-threaded onData() hangs sometimes

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

Hi Gregor,

OK, some testing results:

- last test on production server with 2 incoming DSL connections:
  - 400MByte logfile in non-threaded mode and no hang this time :-/

- local test:
  - in non-threaded mode: hang after a few seconds
  - in multi-threaded mode (default settings, no special worker pool, etc.):
no hang

My assumption would be: fast connection (high data rate, low latency) and
non-threaded
mode in combination with SSL causes some dead lock.

JConsole told me "BLOCKED" for two of three dispatcher Threads at the same
position:
Name: xDispatcherSrv443#1
State: BLOCKED on org.xsocket.connection.IoSSLProcessor@16ac92e owned by:
xDispatcherSrv443#2
Total blocked: 1.822  Total waited: 0

Stack trace: 
org.xsocket.connection.IoActivateableSSLHandler.flushOutgoing(IoActivateable
SSLHandler.java:170)
org.xsocket.connection.IoActivateableSSLHandler.write(IoActivateableSSLHandl
er.java:161)
org.xsocket.connection.NonBlockingConnection.internalFlush(NonBlockingConnec
tion.java:1290)
org.xsocket.connection.NonBlockingConnection.onWriteDataInserted(NonBlocking
Connection.java:1166)
org.xsocket.connection.AbstractNonBlockingStream.write(AbstractNonBlockingSt
ream.java:1109)

That is:

 public void flushOutgoing() throws IOException {
  if (mode == Mode.SSL) {
   synchronized (sslProcessor) {
    if (sslProcessor.isHandshaking()) { //<< HERE
...

just calling:

org.xsocket.connection.IoSSLProcessor
<mailto:org.xsocket.connection.IoSSLProcessor@16ac92e> : synchronized
boolean isHandshaking()

using the "always-SSL" mode tracks the problem down here - ending up in the
same function call:

Name: xDispatcherSrv443#1
State: BLOCKED on org.xsocket.connection.IoSSLProcessor@9b04ac owned by:
xDispatcherSrv443#2
Total blocked: 1.859  Total waited: 0

Stack trace: 
org.xsocket.connection.IoSSLHandler.flush(IoSSLHandler.java:192)
org.xsocket.connection.IoSSLHandler.write(IoSSLHandler.java:187)
org.xsocket.connection.NonBlockingConnection.internalFlush(NonBlockingConnec
tion.java:1290)
org.xsocket.connection.NonBlockingConnection.onWriteDataInserted(NonBlocking
Connection.java:1166)
org.xsocket.connection.AbstractNonBlockingStream.write(AbstractNonBlockingSt
ream.java:1109)

Seems to be a race condition.
I'm filing a bug.

The long-term test in multi-threaded mode is still running.

Thanks for your support so far. :-)

Best regards,
Christian

PS: the last log-entries are:

26.05.2008 14:09:35 org.xsocket.connection.IoSSLProcessor decrypt
FEIN: 1890 decrypted data: 02 e0 02 52 00 21 00 20 00 00 00 05 1a f9 7c 03
19 7d 80 4e 19 85 d0 2e 99 d6 0f f0 1a 19 85 02 39 85 f0 0e 99 d6 0f f0 0e
39 85 99 [...snipped...]
e9 00 b6 ff f9 00  [...output has been cut]
26.05.2008 14:09:35 org.xsocket.connection.AbstractMemoryManager newBuffer
FEIN: allocating 16 kb direct memory
26.05.2008 14:09:35 org.xsocket.connection.ReadQueue append
FEIN: buffer added. new read queue size = 1890
26.05.2008 14:09:35 org.xsocket.connection.IoSynchronizedMemoryManager
recycleMemory
FEIN: recycling 16 kb
26.05.2008 14:09:35 org.xsocket.connection.IoSSLProcessor decrypt
FEIN: 18 decrypted data: 05 00 02 dc 00 0b 05 00 02 da 00 10 05 00 02 da 00
11 
26.05.2008 14:09:35 org.xsocket.connection.ReadQueue append
FEIN: buffer added. new read queue size = 18

  _____  

Von: Gregor Roth [mailto:gre...@go...] 
Gesendet: Montag, 26. Mai 2008 13:19
An: xSo...@li...
Cc: Christian Kahlo
Betreff: Re: [xSocket-develop] Question: SSL with non-threaded onData()
hangs sometimes

Hi Christian,

have you already tested your code in the multithreaded mode? 

If you call the connection methods only within a call back method (thread)
such as onData() or onConnect(), you will not have to synchronize method
calls on the Connection class. xSocket performs the call back methods always
in a serialized for a given connection. 

Independent of this, SSL should work in a non threaded mode as well as in a
multi threaded mode. It is recommended to set the FlushMode to ASYCN to
avoid deadlocks. Did you done this?

If this doesn't help, you should file a bug. Please add also a JUnit test
based on your code.

Gregor 

2008/5/26 Christian Kahlo <ck...@vx...>:

Hi there,

at first thanks for the good work. I stumbled over xSocket and
instantly loved it. ;)

So, I was starting to write a repeater service using SSL.
That means a client and a server both connect the repeater and
identify themselves either as being client or server and to whom
they belong. The result is a pair of two connections - similar to
the proxy example. (it's all about remote desktop stuff)

Everything is working fine - until some sporadic hang occurs in the
onData handler. Because it is non-threaded the whole server hangs.

The code:

       public boolean onData(INonBlockingConnection c)
               throws IOException, BufferUnderflowException,
MaxReadSizeExceededException {
               Node n = (Node)c.getAttachment();

               if(n != null && n.sessionID != null) {
                       Node n2 = n.server ? viewers.get(n.sessionID) :
servers.get(n.sessionID);

                       if(n2 != null) {
                               System.out.print((n2.server ? "S" : "C") +
c.available() + ">");
                               // before hang #1
                               ByteBuffer[] data =
c.readByteBufferByLength(c.available());

                               System.out.print("<" + (n2.server ? "S" :
"C"));
                               // before hang #2
                               n2.c.write(data);

                               System.out.println("#");
                       }
               }

               return true;
       }

I identified two positions where the code may hang. The write hangs far more
often than the
read. But as I said - sometimes it's running for hours, even days. In
another case it died
within 3 minutes. :(

Could anyone please give me a hint what to watch out for? What to tune?
I supposed some synchronisation cludge with the SSL stack and modified
xSocket, so that I
can deliver my own customized SSLEngine. I forced the connection to use RC4
stream-cipher,
because I thought there may be some length difference issue due to
padding/unpadding when
using a block-cipher - but nothing changed. :-/

Or do I simply _have to_ use multi-threading as soon as I want to have SSL?
Did I understand the other threads correctly, that I would have to
synchronize on the
connection objects for each read and write operation when using
multi-threading?

Thanks in advance.

Best regards,
Christian

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
xSocket-develop mailing list
xSo...@li...
https://lists.sourceforge.net/lists/listinfo/xsocket-develop