disconnects and LDAPConnectionPoolStatistics

[Robert Barretto]

Hi Neil,

   I have a question about LDAPConnectionPoolStatistics and the disconnect handler for a LDAPConnectionPool.  Basically, I'm creating a connection pool with one server and five connections in it. I assign a .setDisconnectHandler(…) on the pool so that I get notified when a connection closes. 

Now… If I kill my ldap server, I get five callbacks to handleDisconnect() which is what I would expect. In my disconnect handler I'm not really doing anything except printing out information about the disconnect plust I'm getting the LDAPConnectionPoolStatistics for the pool and printing those out. So basically, I see the following log 5 times:

Disconnection Notification received.
   [Method Name: handleDisconnect]
      [LDAPConnection: LDAPConnection(connected to 192.168.1.11:389)]
      [Server Name: 192.168.1.11]
      [Port: 389]
      [DisconnectType: SERVER_CLOSED_WITHOUT_NOTICE]
      [message: null]
      [Exception: null]
      [Expected: false]
         [Connection Pool Stats: LDAPConnectionPoolStatistics(numAvailableConnections=5, maxAvailableConnections=10, numSuccessfulConnectionAttempts=5, numFailedConnectionAttempts=0, numConnectionsClosedDefunct=0, numConnectionsClosedExpired=0, numConnectionsClosedUnneeded=0, numSuccessfulCheckouts=1, numFailedCheckouts=0, numReleasedValid=1)]

My question is that I was wondering why the 'numAvailableConnections=5' doesn't decrement as the connections are being closed out.  I would have thought I would have seen the stats decrementing over time, so something like:

   ...,numAvailableConnections=4,...
   ...,numAvailableConnections=3,...
   ...,numAvailableConnections=2,...
   ...,numAvailableConnections=1,...
   ...,numAvailableConnections=0,...

In the case where there is only a single server in the connection pool, I'll get an LDAPException when I try to make the next query and at that point the connection stats will show numAvailableConnections=0.  However, if I have multiple servers in my connection pool I'll never know that one server in the pool is down because the connection stats are for the entire pool and not per-server, correct?

So, are LDAPConnectionPoolStatistics not suppose to update during disconnect events, or is there a bug in the way I'm trying to use them?

Thanks,
//Robert

[Neil Wilson]

It's kind of difficult to answer this, because it depends on a number of factors (including how the connection was discovered to be closed.

Ultimately, the value returned by the LDAPConnectionPoolStatistics.getNumAvailableConnections method is the number of connections currently held in the availableConnections field of the LDAPConnectionPool, which is a LinkedBlockingQueue.  When a connection is determined to be invalid for one reason or another (e.g., by a health check, by an error encountered while attempting to process an operation in the pool, or by calling the pool's releaseDefunctConnection method), then the existing connection will be closed and the pool will try to create a new connection to take its place.

If the connection closure is detected during by a periodic health check, then that health check will be a single-threaded process, so it's not unexpected to see the same number of available connections for each call if the pool is able to successfully create a replacement connection (whether to the same server or a different server).  On the other hand, if the reconnect attempt fails then I would expect the number to decrease.

If the connection closure is detected by an exception while using the pool or you explicitly call the releaseDefunctConnection method, then that may be multithreaded and the value you see for the number of available connections will depend on how far each has gotten in the process of replacing the defunct connection.  If none of the re-establish attempts are successful, or if they are still pending, then it's understandable to see no remaining connections.

[Robert Barretto]

Ok. That lines up to what I was seeing.  When I specifically request a connection and I get an LDAPException because there's no connections I get the true value from the connection pool statistics object.  It was only when the connection closure came without me doing an operation (i.e., something the remote server did and not in response to me trying to allocate or use a connection from the pool).

Thanks!
//Robert

[Zbyszek]

How did you discover that Server closed connection?
java.net.Socket.isConnected() method does not seem to work on my environment (Linux RedHat).

I have added my customized DisconnectHandler through LDAPConnectionOptions which simply logs the event, but it does not work when I stop Ldap Server.

[Neil Wilson]

Unfortunately, Java sockets generally can't notice that a connection is no longer established until you try to use them.

If you're not using synchronous mode (which is the default), then there will be a background thread for each connection that is dedicated to reading data from the server, and that read should immediately detect a "normal" socket closure (in which a TCP FIN or RST packet is sent to provide notification of the closure), although it can't detect cases in which that doesn't happen (e.g., as a result of a network outage, or a crash or hang of the system on which the server is running).

If you're using synchronous mode (which offers better performance if you're not going to be performing asynchronous operations), then things are a little different because there's no attempt to read from or interact with a socket until you try to process an operation on that connection.  In the case of a normal socket closure, the SDK would detect the closure immediately and throw an exception right away, but the same problem exists with non-graceful closure.

In general, the best experience may be to use a connection pool rather than individual connections.  This can offer two possibilities.  First, it can use health periodic checking with an option to actually try to process operations on idle connections so that failures can be detected right away.  Second, it can be configured to retry operations that fail in a way that indicates the connection may no longer be valid, so even if a connection has become invalid then the SDK can create a new connection (and if you're using a server set for the connection pool then the connection may potentially be to a different server) and re-try the operation on it.

Neil

[Zbyszek]

Thanks for explanation.
I am using LDAPConnectionPool and synchronous mode.
I am creating pool with fixed number of connections and want to be sure that this initial number will be preserved. I need to almost immediately (e.g. at the time healthcheck tread is being run) report that remote LDAP server has closed connection from its side (but unfortunately without sending any packets).
So, if it is a general problem with handling sockets in Java, why socket.isConnected() is used to validate connection state? And because this assumption is not reliable, healthcheck thread may work incorrectly.
Thanks,
Zbyszek

[Neil Wilson]

I still think that there is value in using isConnected, because if it returns false, then we know that the connection is not valid.

However, it's also the case that for connection pools with connections using synchronous mode, health checking isn't as effective as it could be if the pool isn't configured to try to actually perform LDAP operations.  I thought about this a little, and have come up with a solution that should work for connections in synchronous mode.  I have just committed a change that will cause the pool to try to perform a read with a very short timeout on connections operating in synchronous mode.  If the connection has been closed, or if an unsolicited notification is waiting on that connection, then this will cause it to be noticed right away (at least, on the next health check interval) rather than when the connection is next checked out for use.  This feature is enabled by default, although it can be disabled if it is not desired or causes some unforeseen problem.

I've done some manual testing of this, and it does allow the LDAPConnectionPool to detect connection closure and unsolicited notifications promptly without the need for any client-initiated operations.  Because of the way the LDAPThreadLocalConnectionPool is implemented, this change cannot be used for that pool.

Neil

[Zbyszek]

I have applied your changes (note - LDAPMessages.java needs also be updated).
Your solution fulfills my requirements - thanks a lot!

[Neil Wilson]

Could you clarify what you mean by "LDAPMessages.java needs also be updated"?

Neil

[Zbyszek]

When I tried to build revision 324 from trunk, my compiler couldn't found symbols ERR_POOL_HEALTH_CHECK_CONN_CLOSED, ERR_POOL_HEALTH_CHECK_READ_FAILURE.
And their definition in enum LDAPMessages resolved this issue (I know, there is ldap.properties where apropriate messages are defined, but this didn't work in my env, maybe I did sth wrong - that is not important for me know).