python-ldap-dev

Too many established connections

[Ron T. <Ro...@US...>]

Hello,

I'm running into a problem with python open ldap connections.  It appears
that they are not closing properly.  I'm doing a bind->call->unbind on every
call.  I am also using the ReconnectLdapObject to try to help with "can not
connect" problems, although I still get a significant number of can not
connect problems that I handle with a retry.

The server died the first time we tried to put a significant load on it.

Does anyone have suggestions for fixing this problem or figuring out what is
leaving connections open?

Thanks,

Ron Teitelbaum

Re: Too many established connections

[Michael S. <mi...@st...>]

Ron Teitelbaum wrote:
> 
> I'm running into a problem with python open ldap connections. 
                                   ^^^^^^^^^^^^^^^^
Do you mean python-ldap connections or connections to an OpenLDAP server 
from Python?

> It appears that they are not closing properly.

Can you check in the server's log whether unbind is processed?

> I'm doing a bind->call->unbind on every
> call.

Is that really necessary?

>  I am also using the ReconnectLdapObject to try to help with "can not
> connect" problems,

Well, using ReconnectLdapObject and doing bind-search-unbind all the time is 
somewhat contradictory since ReconnectLdapObject is for keeping long-lasting 
connections alive. But this should not be the problem.

> although I still get a significant number of can not
> connect problems that I handle with a retry.

Any network problems in between?

> The server died the first time we tried to put a significant load on it.

What server (vendor and version) is this? Which version of python-ldap and 
which version OpenLDAP libs are used?

Ciao, Michael.

CLOSE_WAIT sessions (was: Too many established connections)

[Ron T. <Ro...@US...>]

Hi Michael,

> -----Original Message-----
> From: Michael Ströder 
> 
> Ron Teitelbaum wrote:
> >
> > I'm running into a problem with python open ldap connections.
>                                    ^^^^^^^^^^^^^^^^
> Do you mean python-ldap connections or connections to an OpenLDAP server
> from Python?

Yes python-ldap.  I'm using a smalltalk -> python-ldap 2.2.1 -> openLDAP
2.3-2.3.30 on Ubuntu 7.04 (GNU/Linux 2.6.20-16-generic)

> 
> Can you check in the server's log whether unbind is processed?

I went through and found that there was some exception handling in my code
that caused the unbind to be skipped.  I fixed that and things are
significantly better now.  Thank you!

I am still having a problem but it looks like we had ldap hang up.  There
were some connections with CLOSE_WAIT that caused everything to get hung up.
It looked like we were not able to bind after that.  Have you seen that
problem before?  I rebooted the system and have not seen the error again, so
things are currently stable, we are going to try a load test in about an
hour or so.

> 
> > I'm doing a bind->call->unbind on every
> > call.
> 
> Is that really necessary?

When I started writing this code I assumed that I needed some sort of
connection pooling.  When bind->call->unbind turned out to be very fast I
figured the pool was a premature optimization.  Maybe I should have stuck
with my gut feeling.

> 
> >  I am also using the ReconnectLdapObject to try to help with "can not
> > connect" problems,
> 
> Well, using ReconnectLdapObject and doing bind-search-unbind all the time
> is
> somewhat contradictory since ReconnectLdapObject is for keeping long-
> lasting
> connections alive. But this should not be the problem.

That's interesting.  I thought ReconnectLdapObject's primary purpose was to
help with can-not-connect issues.  Very often we get can-not-connect errors.
Sometimes we get them even when things worked.  We wrote some handlers to
retry and to ignore errors associated with duplicate processing.  Any idea
why we would be getting can-not-connect errors?  (we get them sporadically
with bind, write, search ...) 

> 
> > although I still get a significant number of can not
> > connect problems that I handle with a retry.
> 
> Any network problems in between?

No for right now we are running OpenLdap and Python-ldap and Smalltalk all
on the same box.
 
> 
> > The server died the first time we tried to put a significant load on it.
> 
> What server (vendor and version) is this? Which version of python-ldap and
> which version OpenLDAP libs are used?

python-ldap 2.2.1 -> openLDAP 2.3-2.3.30 on Ubuntu 7.04 (GNU/Linux
2.6.20-16-generic)

The older releases above are to stay inline with the Debian release.  Also
we are currently locked into python 2.4.4.  Upgrading python is not
currently an option for this system.

> 
> Ciao, Michael.

Thank you very much for your help!

Ron Teitelbaum

Re: CLOSE_WAIT sessions

[Michael S. <mi...@st...>]

Ron,

Ron Teitelbaum wrote:
> Yes python-ldap.  I'm using a smalltalk -> python-ldap 2.2.1 -> openLDAP
> 2.3-2.3.30 on Ubuntu 7.04 (GNU/Linux 2.6.20-16-generic)

There has been important fixes for Python 2.5 and also ReconnectLDAPObject 
in 2.3.0. So I'd recommend to build python-ldap 2.3.2 (released today).

>>> I'm doing a bind->call->unbind on every
>>> call.
>> Is that really necessary?
> 
> When I started writing this code I assumed that I needed some sort of
> connection pooling.  When bind->call->unbind turned out to be very fast I
> figured the pool was a premature optimization.  Maybe I should have stuck
> with my gut feeling.

A connection pool and making a new connection all the time are very extreme 
variants. I don't know your application's needs though.

>>>  I am also using the ReconnectLdapObject to try to help with "can not
>>> connect" problems,
>> Well, using ReconnectLdapObject and doing bind-search-unbind all the time
>> is
>> somewhat contradictory since ReconnectLdapObject is for keeping long-
>> lasting
>> connections alive. But this should not be the problem.
> 
> That's interesting.  I thought ReconnectLdapObject's primary purpose was to
> help with can-not-connect issues.

Yes, but mainly for long-lasting connections.

>  Very often we get can-not-connect errors.
> Sometimes we get them even when things worked.  We wrote some handlers to
> retry and to ignore errors associated with duplicate processing.  Any idea
> why we would be getting can-not-connect errors?  (we get them sporadically
> with bind, write, search ...) 

You mean ldap.SERVER_DOWN exceptions? Yes, you could ReconnectLdapObject to 
make things a litte easier within your application. If you're solely using 
the synchronous methods you don't have to catch the exceptions yourself. But 
if you experience this to happen very often you should try to solve the 
problem causing this.

>>> The server died the first time we tried to put a significant load on it.
>> What server (vendor and version) is this? Which version of python-ldap and
>> which version OpenLDAP libs are used?
> 
> python-ldap 2.2.1 -> openLDAP 2.3-2.3.30 on Ubuntu 7.04 (GNU/Linux
> 2.6.20-16-generic)
> 
> The older releases above are to stay inline with the Debian release.

I see several problems with Debian packages:

1. Some Debian packages of python-ldap were heavily patched to work with 
ancient OpenLDAP 2.1 libs. I refuse to give support for those. Don't know 
about the particular version you're using though.

2. OpenLDAP debian packages are linked against gnu-tls which causes all 
sorts of problems when LDAP over SSL (LDAPS) or StartTLS ext.op. is used. 
This could lead to ldap.SERVER_DOWN to be raised if anything goes wrong at 
SSL/TLS level

Ciao, Michael.