[beep4j-user] Concurrency issues

[Thomson, M. <Mar...@an...>]

Hi,

I'm seeing a few problems with beep4j under high loads in concurrent
applications.


~ Out of order messages

On inspection of the code in SessionImpl.java, it appears that out of
order receipt of response messages causes a protocol exception (which
drops the connection).  The replyListeners collection doesn't maintain
the set of ReplyListener instances for each channel based on the message
number - the collection is a list.  When a response arrives it pulls the
first item of the list and raises the exception if the number doesn't
match.

One of the main advantages that BEEP provides is the out of order
receipt of responses.  If this feature wasn't available, we might as
well be using HTTP.

I have a small patch that changes SessionImpl.java from 0.1.1 that uses
a Map rather than a List, i.e.
    Map<Integer, Map<Integer, ReplyListenerHolder>> replyListeners

I can send out that patch; however, it appears that there are a large
number of changes between 0.1.1 and what is on the SVN trunk.  I am
concerned that the changes don't fix this and could cause more problems.

It's difficult to follow the thread of execution; but based on a rough
inspection, I see that the ReplyListener is now saved in
FilterChainTargetHolder, which uses a ThreadLocal.  This would present a
problem if there were two requests sent from the same thread; the second
(and any subsequent requests from that thread) would overwrite the
ReplyListener from the first.


~ Impact of congestion control for TCP mapping

We are running some tests using a single BEEP channel to send a
moderately high rate of messages between two applications.  Both are
using beep4j.  Both do non-trivial processing on multiple threads.  We
frequently get errors with seqno mismatches.

The following (abridged) sequence is direct from a Wireshark packet
capture:

RPY 3 12 * 3784 312
<!-- first part of reply to 3-12 -->
END
RPY 3 9 * 4257 0
END
SEQ 3 23232 4096
RPY 3 9 . 4257 473
<!-- reply to 3-9 -->
END
RPY 3 12 . 4096 161
<!-- second part of reply to 3-12 -->
END
RPY 3 6 . 5203 473
<!-- reply to 3-6 -->
END

As you can see - out of order message numbers on replies (I'm using the
SessionImpl.java patch mentioned above).  More alarming is the fact that
beep4j sends the first part of RPY 3-9 before finishing RPY 3-12.  What
is causes the failure is the seqno mismatch it sees when the first
(empty) part of RPY 3-9 is found.

I think that the problem is that when there are two frames in the queue
and the sender window (i.e. that advertised by the other end) is too
small, TCPMapping splits the first frame (correctly), puts the remainder
back on the queue (correctly), but then continues to try to process
subsequent frames.  It should not be sending anything more (except maybe
the SEQ message) when it does not have any space left in the window.

The following might fix that part:

	private Frame nextFrame() {
+		if (frames.isEmpty() || senderWindow.remaining() <= 0) {
-		if (frames.isEmpty()) {
		    return null;

But I'm not sure what the calling context expects from sendFrames.  I
expect that it should manage, but I'll need to test this further to find
out.


On the other hand, the following looks like a problem, but it might
actually work.  When TCPMapping receives a SEQ message it immediately
calls transport.sendBytes(createSEQFrame(channel, ackno, windowSize))
rather than creating and queuing up a new Frame object.

I understand how this might be desirable, but it might be better (for
the sake of interoperability) to defer the sending of this message if
there is an outstanding split frame.  From RFC 3080:

   When a message is segmented and sent as several frames, those frames
   must be sent sequentially, without any intervening frames from other
   messages on the same channel.

I have a few partially formed thoughts on how to solve this, but I'll
get back to that tomorrow.  I'll test the simple senderWindow check once
I have both fixes together.


Let me know if anyone wants the SessionImpl.java patch for 0.1.1.

Cheers,
Martin 

------------------------------------------------------------------------------------------------
This message is for the designated recipient only and may
contain privileged, proprietary, or otherwise private information.  
If you have received it in error, please notify the sender
immediately and delete the original.  Any unauthorized use of
this email is prohibited.
------------------------------------------------------------------------------------------------
[mf2]