Re: [ldap-sdk-discuss] LDAP connection pool behaviour

[Vladimir D. <vla...@co...>]

Thanks Neil for your comprehensive response. You need to digest it
first, and then we'll get back to you.

Vladimir

On 12/05/16 09:05, Neil A. Wilson wrote:
> Sorry for the late response. My comments are inline below.
>
> Neil
>
>
> On 05/11/2016 05:40 AM, Vladimir Dzhuvinov wrote:
>> Hi,
>>
>> I would like to report on some recently observed behaviour of the LDAP
>> connection pool in v3.3.1.
> Does it seem that the 3.3.1 version is behaving differently than the
> previous version that you were using, or is it possible that this
> behavior also exists in older versions?
>
> The only change to the connection pool implementation between the 3.3.0
> and 3.3.1 releases is a very minor update to fix a bug that prevented
> connection pool statistics from recording a failed attempt to establish
> a new connection in certain circumstances, and I can't see how that
> change would lead to any case in which the number of connections in the
> pool grows.
>
> There were a couple of changes between the 3.0.0 and 3.1.0 releases, but
> I've reviewed those changes and none of them seem to have the potential
> to cause a regression like what you're reporting.
>
>
>
>> The setup looks like this:
>>
>> 8 LDAP clients, each on the node of a clustered web app.
>>
>> Each LDAP client is configured with an LDAP connection pool:
>>
>> * Initial number of connections: 0
>> * Max number of connections: 1024
>> * Max connection wait time: 500ms
>> * Max connection age: 0
>>
>>
>> LDAP requests are run against the pool, i.e. connections are not checked
>> out, then returned.
>>
>>
>> With this setup and the typical app load the number of available
>> connections settles to ~200. On several occasions however, when a new
>> LDAP client / app node is started, the pool is slow to create new
>> connections, settling to about 5 to 10, and leading to a degraded app
>> performance. To address this, the setting for the initial number of
>> connections was raised to 50. With this setup however, the pooled
>> connections immediately jumped to the max limit (~1000).
> It doesn't make sense that merely changing the initial number of
> connections would cause the total number of connections to increase.
> Once the initial connections have been established, the pool should only
> create new connections if one is needed when all of the existing
> connections are already in use.
>
> At least in theory, the only way that you should end up with 1000
> connections established as part of the pool is to have a situation in
> which you have 1000 threads in your application all trying to process
> LDAP operations at the same time. Do you think that it's reasonable that
> your application might be processing 1000 LDAP requests simultaneously?
>
> Note that the number of connections in the pool won't go down on its
> own. You either need to call discardConnection or shrinkPool to have the
> pool get rid of connections. In all other circumstances, if a connection
> needs to be closed then the pool will try to replace it with a
> newly-established connection. So if you have 1000 connections, then that
> doesn't mean that all 1000 of them are busy right now, but it should
> indicate that 1000 simultaneous connections were needed at some point in
> the life of the pool.
>
> Have you looked at the connection pool statistics to see if its numbers
> match what you're seeing?  If a pool has 1000 connections, then
> getNumSuccessfulConnectionAttempts should return a value that is 1000
> more than the sum of the values returned by the
> getNumConnectionsClosedDefunct, getNumConnectionsClosedExpired, and
> getNumConnectionsClosedUnneeded methods.
>
>
>
>> My question: is there any recipe or configuration strategy to ensure the
>> pooled connection count is a good match for the LDAP request
>> requirements of the client?
> Unless your client is doing some very weird things, you shouldn't need
> more than one connection per application thread.  And the only scenarios
> I can think of that might be able to allocate more than one connection
> per thread would require you to check out and release connections
> yourself, but it sounds like you're not doing that.
>
>
>
>> We also ran a few simple tests that showed different connection build up
>> with checked out LDAP connections, vs running requests against the pool.
>> What is the preferred way to run LDAP requests for stable performance
>> and optimal utilisation of the pooled connections?
> I'm not sure why you would see any difference in the accumulation of
> connections when you let the pool check out and release connections in
> the course of processing an operation and when you check out and release
> the connections for yourself.  Letting the pool do it for you enables
> the potential for using additional functionality that you can't get when
> checking out and restoring the connections for yourself (for example,
> automatically retrying a failed operation on a newly-established
> connection), and also simplifies your application code, so I would
> recommend going with that approach.  But ultimately, if you have the
> pool check out and release connections for you, then it should be using
> exactly the same methods to check out and release connections that you
> would use if you were doing it yourself.
>
> Do you have any kind of reproducible test case that you could provide to
> demonstrate the problem?  If not, would you be able to provide the code
> that you're using to create and manage the connection pool (in
> particular, the constructor you're using, along with any calls to setter
> methods for customizing the pool behavior, like enabling retry, setting
> a maximum connection age, configuring a health check, etc.)?  Feel free
> to obscure or hide any sensitive information, since it's more about
> understanding what calls you're making than exactly what arguments are
> provided.
>
>
>
>>
>> Any suggestions or insights will be appreciated.
>>
>> Thanks,
>>
>>
>> Vladimir
>>
>
>
>
> ------------------------------------------------------------------------------
> Mobile security can be enabling, not merely restricting. Employees who
> bring their own devices (BYOD) to work are irked by the imposition of MDM
> restrictions. Mobile Device Manager Plus allows you to control only the
> apps on BYO-devices by containerizing them, leaving personal data untouched!
> https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
>
>
> _______________________________________________
> ldap-sdk-discuss mailing list
> lda...@li...
> https://lists.sourceforge.net/lists/listinfo/ldap-sdk-discuss

-- 
Vladimir Dzhuvinov :: vla...@co...