Possible TCP/IP connections saturation
LIXA, LIbre XA, is a free and open source XA transaction manager
Brought to you by:
tiian
Menu
▾
▴
#19 Possible TCP/IP connections saturation
A client that performs a lot of tx_open/tx_close can saturate all the available TCP/IP connection and "connect" functions fail.
SO_LINGER could be a "solution", but this may generate different issues as W. Richard Stevens explained in "UNIX network programming".
This is a potential bug and it could be exploited in a stressed production environment; the "solution" to this type of problem is not very good, so the implementation of the "solution" is delayed until someone really exploits the issue.
I know this is a potential issue, this bug will be stay open long time (I suppose).
Christian, do you think this could also be a problem if an AP does a lot of tx_begin()/tx_commit(),tx_rollback() calls?
At the moment, I am running some AP benchmarking, and found that lixad is kept very busy, which is expected, but could it be related?
What was your proposed "solution" for this? Have you thought of any other means of communication between the TM (lixac) and state server (lixad)?
The typical outcome of this time of issue is related to high number of sockets in status TIME_WAIT, until no more available connections are available. From a point to point, typically 65k.
Here's a very clear explanation of the issue: http://stackoverflow.com/questions/3757289/tcp-option-so-linger-zero-when-its-required
In normal condition, LIXA procotol close the socket starting from the client peer and the issue should not appear.
Anyway, a "very busy lixad" could be the symptom of an high number of disk synchronizations. To understand a possible disk sync performance issue, you can trace module LIXA_TRACE_MOD_SERVER_STATUS and look at trace records produced by function "thread_status_sync_files": two records are really interesting:
because you can retrieve the real latency introduced by disk synchronization.
Disk synchronization can be mitigated playing with 2 parameters as explained here: http://lixa.sourceforge.net/lixa-doc/html/ch09.html, but they are designed most for scalability than for single thread peak performance.