Re: [ldap-sdk-discuss] LDAP connection pool behaviour

[Vladimir D. <vla...@co...>]

Hi Neil,

It turned out that the LDAP server was getting bogged down, which then
caused a pile up of requests and connections waiting to be served.
Monitoring pool and server seems crucial here.

The connection pool stats bug was reported by us, thanks for fixing it.
There's no apparent change of behaviour between 3.1.1 and 3.1.0 as seen
from our side.

Thank

Vladimir

On 12/05/16 11:20, Vladimir Dzhuvinov wrote:
> Thanks Neil for your comprehensive response. You need to digest it
> first, and then we'll get back to you.
>
> Vladimir
>
> On 12/05/16 09:05, Neil A. Wilson wrote:
>> Sorry for the late response. My comments are inline below.
>>
>> Neil
>>
>>
>> On 05/11/2016 05:40 AM, Vladimir Dzhuvinov wrote:
>>> Hi,
>>>
>>> I would like to report on some recently observed behaviour of the LDAP
>>> connection pool in v3.3.1.
>> Does it seem that the 3.3.1 version is behaving differently than the
>> previous version that you were using, or is it possible that this
>> behavior also exists in older versions?
>>
>> The only change to the connection pool implementation between the 3.3.0
>> and 3.3.1 releases is a very minor update to fix a bug that prevented
>> connection pool statistics from recording a failed attempt to establish
>> a new connection in certain circumstances, and I can't see how that
>> change would lead to any case in which the number of connections in the
>> pool grows.
>>
>> There were a couple of changes between the 3.0.0 and 3.1.0 releases, but
>> I've reviewed those changes and none of them seem to have the potential
>> to cause a regression like what you're reporting.
>>
>>
>>
>>> The setup looks like this:
>>>
>>> 8 LDAP clients, each on the node of a clustered web app.
>>>
>>> Each LDAP client is configured with an LDAP connection pool:
>>>
>>> * Initial number of connections: 0
>>> * Max number of connections: 1024
>>> * Max connection wait time: 500ms
>>> * Max connection age: 0
>>>
>>>
>>> LDAP requests are run against the pool, i.e. connections are not checked
>>> out, then returned.
>>>
>>>
>>> With this setup and the typical app load the number of available
>>> connections settles to ~200. On several occasions however, when a new
>>> LDAP client / app node is started, the pool is slow to create new
>>> connections, settling to about 5 to 10, and leading to a degraded app
>>> performance. To address this, the setting for the initial number of
>>> connections was raised to 50. With this setup however, the pooled
>>> connections immediately jumped to the max limit (~1000).
>> It doesn't make sense that merely changing the initial number of
>> connections would cause the total number of connections to increase.
>> Once the initial connections have been established, the pool should only
>> create new connections if one is needed when all of the existing
>> connections are already in use.
>>
>> At least in theory, the only way that you should end up with 1000
>> connections established as part of the pool is to have a situation in
>> which you have 1000 threads in your application all trying to process
>> LDAP operations at the same time. Do you think that it's reasonable that
>> your application might be processing 1000 LDAP requests simultaneously?
>>
>> Note that the number of connections in the pool won't go down on its
>> own. You either need to call discardConnection or shrinkPool to have the
>> pool get rid of connections. In all other circumstances, if a connection
>> needs to be closed then the pool will try to replace it with a
>> newly-established connection. So if you have 1000 connections, then that
>> doesn't mean that all 1000 of them are busy right now, but it should
>> indicate that 1000 simultaneous connections were needed at some point in
>> the life of the pool.
>>
>> Have you looked at the connection pool statistics to see if its numbers
>> match what you're seeing?  If a pool has 1000 connections, then
>> getNumSuccessfulConnectionAttempts should return a value that is 1000
>> more than the sum of the values returned by the
>> getNumConnectionsClosedDefunct, getNumConnectionsClosedExpired, and
>> getNumConnectionsClosedUnneeded methods.
>>
>>
>>
>>> My question: is there any recipe or configuration strategy to ensure the
>>> pooled connection count is a good match for the LDAP request
>>> requirements of the client?
>> Unless your client is doing some very weird things, you shouldn't need
>> more than one connection per application thread.  And the only scenarios
>> I can think of that might be able to allocate more than one connection
>> per thread would require you to check out and release connections
>> yourself, but it sounds like you're not doing that.
>>
>>
>>
>>> We also ran a few simple tests that showed different connection build up
>>> with checked out LDAP connections, vs running requests against the pool.
>>> What is the preferred way to run LDAP requests for stable performance
>>> and optimal utilisation of the pooled connections?
>> I'm not sure why you would see any difference in the accumulation of
>> connections when you let the pool check out and release connections in
>> the course of processing an operation and when you check out and release
>> the connections for yourself.  Letting the pool do it for you enables
>> the potential for using additional functionality that you can't get when
>> checking out and restoring the connections for yourself (for example,
>> automatically retrying a failed operation on a newly-established
>> connection), and also simplifies your application code, so I would
>> recommend going with that approach.  But ultimately, if you have the
>> pool check out and release connections for you, then it should be using
>> exactly the same methods to check out and release connections that you
>> would use if you were doing it yourself.
>>
>> Do you have any kind of reproducible test case that you could provide to
>> demonstrate the problem?  If not, would you be able to provide the code
>> that you're using to create and manage the connection pool (in
>> particular, the constructor you're using, along with any calls to setter
>> methods for customizing the pool behavior, like enabling retry, setting
>> a maximum connection age, configuring a health check, etc.)?  Feel free
>> to obscure or hide any sensitive information, since it's more about
>> understanding what calls you're making than exactly what arguments are
>> provided.
>>
>>
>>
>>> Any suggestions or insights will be appreciated.
>>>
>>> Thanks,
>>>
>>>
>>> Vladimir
>>>
>>
>>
>> ------------------------------------------------------------------------------
>> Mobile security can be enabling, not merely restricting. Employees who
>> bring their own devices (BYOD) to work are irked by the imposition of MDM
>> restrictions. Mobile Device Manager Plus allows you to control only the
>> apps on BYO-devices by containerizing them, leaving personal data untouched!
>> https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
>>
>>
>> _______________________________________________
>> ldap-sdk-discuss mailing list
>> lda...@li...
>> https://lists.sourceforge.net/lists/listinfo/ldap-sdk-discuss
>
>
> ------------------------------------------------------------------------------
> Mobile security can be enabling, not merely restricting. Employees who
> bring their own devices (BYOD) to work are irked by the imposition of MDM
> restrictions. Mobile Device Manager Plus allows you to control only the
> apps on BYO-devices by containerizing them, leaving personal data untouched!
> https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
>
>
> _______________________________________________
> ldap-sdk-discuss mailing list
> lda...@li...
> https://lists.sourceforge.net/lists/listinfo/ldap-sdk-discuss

-- 
Vladimir Dzhuvinov :: vla...@co...