Menu
▾
▴
python-ldap-dev
|
From: Leif H. <lei...@pr...> - 2002-06-25 19:00:51
|
Well, I don't know if this is the same problem I had with Python LDAP v1.x,
we haven't tested v2.x yet. But, the result() function in Python LDAP can go
into a very tight poll loop, with extreme effects if the Python process is
running on the same machines as the LDAP server. Python will almost
completely starve slapd for any CPU time ...
Adding a short sleep() in the polling loop of ldapobject.result() helps, a
lot. Like
while all:
while ldap_result[0] is None:
if (timeout>=0) and (time.time()-start_time>timeout):
self._ldap_call(self._l.abandon,msgid)
raise _ldap.TIMELIMIT_EXCEEDED(
"LDAP time limit (%d secs) exceeded." % (timeout)
)
ldap_result = self._ldap_call(self._l.result,msgid,0,0)
if ldap_result[0] is None:
time.sleep(.01)
if ldap_result[1] is None:
break
(note the time.sleep() call if there was no result). Alternatively, adding
an (arbitrarily) long timeout in the call to ldap_result() also accomplishes
the same thing, like:
ldap_result = self._ldap_call(self._l.result,msgid,0,15 * 60)
I haven't dug deep into this problem yet, to figure out if this is an
OpenLDAP library problem, or a Python LDAP problem. I just know that
preventing ldapobject.result() from going into the tight polling loop solved
our problems. :-)
Cheers,
-- Leif
|
|
From: Leif H. <lei...@pr...> - 2002-06-27 16:58:52
|
I think you want to do the sleep only if ldap_result[0] is None, otherwise you'll cause a sleep every loop, regardless if data is available or not. That's why I originally put it after the call to result() inside the polling loop. -- Leif |
|
From: <mi...@st...> - 2002-06-28 07:22:50
|
Leif Hedstrom wrote:
> I think you want to do the sleep only if ldap_result[0] is None, otherwise
> you'll cause a sleep every loop, regardless if data is available or not.
???
while ldap_result[0] is None:
if (timeout>=0) and (time.time()-start_time>timeout):
self._ldap_call(self._l.abandon,msgid)
raise _ldap.TIMELIMIT_EXCEEDED(
"LDAP time limit (%d secs) exceeded." % (timeout)
)
time.sleep(0.0001)
ldap_result = self._ldap_call(self._l.result,msgid,0,0)
> That's why I originally put it after the call to result() inside the polling
> loop.
Note that there is a first call to self._l.result() outside both
while-loops and the time.sleep() is in the inner loop (while
ldap_result[0] is None:). IMHO it should be equivalent.
Ciao, Michael.
|
|
From: Leif H. <lei...@pr...> - 2002-06-28 17:34:33
|
> Note that there is a first call to self._l.result() outside both > while-loops and the time.sleep() is in the inner loop (while > ldap_result[0] is None:). IMHO it should be equivalent. Yeah, my bad, you are absolutely correct. I still find it kinda kludgy to have to put a sleep() statement in the loop like this, but it works for us... ;-) ciao, -- leif |
|
From: Jens V. <je...@zo...> - 2002-06-25 20:02:50
|
leif, i will be darned :) your little hack does indeed work on my setup, too. a test script that took 30 seconds to run now finishes in 2 seconds... jens On Tuesday, June 25, 2002, at 03:00 , Leif Hedstrom wrote: > Well, I don't know if this is the same problem I had with Python LDAP v1. > x, > we haven't tested v2.x yet. But, the result() function in Python LDAP can > go > into a very tight poll loop, with extreme effects if the Python process is > running on the same machines as the LDAP server. Python will almost > completely starve slapd for any CPU time ... > > Adding a short sleep() in the polling loop of ldapobject.result() helps, a > lot. Like > > while all: > while ldap_result[0] is None: > if (timeout>=0) and (time.time()-start_time>timeout): > self._ldap_call(self._l.abandon,msgid) > raise _ldap.TIMELIMIT_EXCEEDED( > "LDAP time limit (%d secs) exceeded." % (timeout) > ) > ldap_result = self._ldap_call(self._l.result,msgid,0,0) > if ldap_result[0] is None: > time.sleep(.01) > if ldap_result[1] is None: > break > > (note the time.sleep() call if there was no result). Alternatively, adding > an (arbitrarily) long timeout in the call to ldap_result() also > accomplishes > the same thing, like: > > ldap_result = self._ldap_call(self._l.result,msgid,0,15 * 60) > > I haven't dug deep into this problem yet, to figure out if this is an > OpenLDAP library problem, or a Python LDAP problem. I just know that > preventing ldapobject.result() from going into the tight polling loop > solved > our problems. :-) > > Cheers, > > -- Leif > |
|
From: <mi...@st...> - 2002-06-27 06:47:17
|
Leif Hedstrom wrote:
> Well, I don't know if this is the same problem I had with Python LDAP v1.x,
> we haven't tested v2.x yet. But, the result() function in Python LDAP can go
> into a very tight poll loop, with extreme effects if the Python process is
> running on the same machines as the LDAP server. Python will almost
> completely starve slapd for any CPU time ...
>
> Adding a short sleep() in the polling loop of ldapobject.result() helps, a
> lot.
Yes, you're right. A time.sleep(0.0001) right before the inner
result() call makes python-ldap hand over the CPU to the OS.
> while all:
> while ldap_result[0] is None:
> if (timeout>=0) and (time.time()-start_time>timeout):
> self._ldap_call(self._l.abandon,msgid)
> raise _ldap.TIMELIMIT_EXCEEDED(
> "LDAP time limit (%d secs) exceeded." % (timeout)
> )
> ldap_result = self._ldap_call(self._l.result,msgid,0,0)
> if ldap_result[0] is None:
> time.sleep(.01)
> if ldap_result[1] is None:
> break
It's possible to make it somewhat simpler since we have a first
result() call before the while loops.
while all:
while ldap_result[0] is None:
if (timeout>=0) and (time.time()-start_time>timeout):
self._ldap_call(self._l.abandon,msgid)
raise _ldap.TIMELIMIT_EXCEEDED(
"LDAP time limit (%d secs) exceeded." % (timeout)
)
time.sleep(0.0001)
ldap_result = self._ldap_call(self._l.result,msgid,0,0)
if ldap_result[1] is None:
break
all_results.extend(ldap_result[1])
ldap_result = None,None
return all_results
> Alternatively, adding
> an (arbitrarily) long timeout in the call to ldap_result() also accomplishes
> the same thing, like:
>
> ldap_result = self._ldap_call(self._l.result,msgid,0,15 * 60)
Which is a bad idea in a multi-threaded environment.
> I haven't dug deep into this problem yet, to figure out if this is an
> OpenLDAP library problem, or a Python LDAP problem.
The main problem here is that the OpenLDAP libs are not
thread-safe. Therefore a module-wide lock is needed to serialize
all calls into OpenLDAP libs. But just wrapping ldap_result() with
locking around it would lead to blocking threads if one thread is
waiting for large search results. That's why I implemented
LDAPObject.result() in Python like it is today. Unfortunately we
cannot deal with waiting for data at the socket level which leads
to higher-level polling loop.
Everybody is encouraged to try the time.sleep(0.0001) hack and
look how it "feels" now. BTW: it makes the simple benchmark I
posted yesterday slower. It always depends what you wanna
optimize... ;-)
I suspect this still might not fully explain the
30s-vs.-immediate-results reports. There may be more issues with
other effects like Mauro described.
I guess the best solution would be if somebody with some spare
cycles digs into OpenLDAP's libldap_r to check if it's ready for
use with python-ldap. This would make it possible to do a finer
grained locking on LDAP connections instead of module-wide locking
allowing different threads with different LDAPObject instances to
run without blocking each other.
Ciao, Michael.
|
|
From: Jens V. <je...@zo...> - 2002-06-27 11:45:39
|
On Wednesday, June 26, 2002, at 04:58 , Michael Str=F6der wrote:
> It's possible to make it somewhat simpler since we have a first =
result()=20
> call before the while loops.
>
> while all:
> while ldap_result[0] is None:
> if (timeout>=3D0) and (time.time()-start_time>timeout):
> self._ldap_call(self._l.abandon,msgid)
> raise _ldap.TIMELIMIT_EXCEEDED(
> "LDAP time limit (%d secs) exceeded." % (timeout)
> )
> time.sleep(0.0001)
> ldap_result =3D self._ldap_call(self._l.result,msgid,0,0)
> if ldap_result[1] is None:
> break
> all_results.extend(ldap_result[1])
> ldap_result =3D None,None
> return all_results
>
this simplified version seems to slow down my setup. all of a sudden i =
get=20
only 50-70% of my previous speed. here is a result set with the result=20=
method changed to the format shown above::
*** Read the RootDSE on same connection
50.432497 searches/second
*** Read the RootDSE on newly created connection without extra simple =
bind
49.699850 searches/second
*** Read the RootDSE on newly created connection with an extra simple =
bind
31.885130 searches/second
here's a result set with leif's version::
*** Read the RootDSE on same connection
100.785835 searches/second
*** Read the RootDSE on newly created connection without extra simple =
bind
95.549340 searches/second
*** Read the RootDSE on newly created connection with an extra simple =
bind
49.825837 searches/second
jens
|
|
From: <mi...@st...> - 2002-06-28 07:09:55
|
Jens Vagelpohl wrote: >=20 > On Wednesday, June 26, 2002, at 04:58 , Michael Str=F6der wrote: >=20 >> It's possible to make it somewhat simpler since we have a first=20 >> result() call before the while loops. >> >> while all: >=20 > this simplified version seems to slow down my setup. Yes. I wrote that when posting the code snippet. > here's a result set with leif's version:: Leif's version sets non-zero timeout. Therefore it's faster since=20 OpenLDAP's ldap_result() can use select() to determine=20 just-in-time when data is ready to be read. But it blocks which is=20 a bad thing because of ldap._ldap_lock serializing *all* calls... Just adding the time.sleep() hands over the CPU to the OS. Off=20 course the while-loop is not just-in-time there if received data=20 is ready. Again, the problem is that the OpenLDAP libs are not thread-safe... Ciao, Michael. |
|
From: Joe L. <jl...@op...> - 2002-06-28 16:39:25
|
Michael. We beat this one up a bit before... But I asked Luke Howard (who is = generally in the know on this stuff) again about OpenLDAP thread-safety, = and he said that it is thread safe. Perhaps Leif can chime in here on = where it stands. On Thursday, June 27, 2002, at 10:47 AM, Michael Str=F6der wrote: > Jens Vagelpohl wrote: >> On Wednesday, June 26, 2002, at 04:58 , Michael Str=F6der wrote: >>> It's possible to make it somewhat simpler since we have a first = result() call before the while loops. >>> >>> while all: >> this simplified version seems to slow down my setup. > > Yes. I wrote that when posting the code snippet. > >> here's a result set with leif's version:: > > Leif's version sets non-zero timeout. Therefore it's faster since = OpenLDAP's ldap_result() can use select() to determine just-in-time when = data is ready to be read. But it blocks which is a bad thing because of = ldap._ldap_lock serializing *all* calls... > > Just adding the time.sleep() hands over the CPU to the OS. Off course = the while-loop is not just-in-time there if received data is ready. > > Again, the problem is that the OpenLDAP libs are not thread-safe... > > Ciao, Michael. > > > > ------------------------------------------------------- > This sf.net email is sponsored by:ThinkGeek > Caffeinated soap. No kidding. > http://thinkgeek.com/sf > _______________________________________________ > Python-LDAP-dev mailing list > Pyt...@li... > https://lists.sourceforge.net/lists/listinfo/python-ldap-dev |
|
From: <mi...@st...> - 2002-06-28 16:44:28
|
Joe Little wrote: > > We beat this one up a bit before... But I asked Luke Howard (who is > generally in the know on this stuff) again about OpenLDAP thread-safety, > and he said that it is thread safe. Since when? Which version? Which lib? libldap_r is supposed to be reentrant but that's not what python-ldap is built against. Furthermore it was not possible to get a positive statement from Kurt Zeilenga last time I asked. Ciao, Michael. |
|
From: Joe L. <jl...@op...> - 2002-06-28 16:49:21
|
two snippets from Luke when speaking about 2.1 (and whether it was = finally thread safe: lukeh: On what is thread safe? The client library? The server? I thought it was always pretty much thread safe. I think they've shaken down some issues with the transactional (bdb) backend. When I mentioned complaints here about thread safety, his response: I haven't seen those complaints. The OpenLDAP client library is = guaranteed to be thread safe as long as you (a) link against libldap_r and (b) do not permit concurrent access to a single LDAP session (ie. use one per thread or a mutex around it). So, libldap_r is one requirement... but the other seems to be already = set by our use of internal thread locking to that single session. On Friday, June 28, 2002, at 09:43 AM, Michael Str=F6der wrote: > Joe Little wrote: >> We beat this one up a bit before... But I asked Luke Howard (who is = generally in the know on this stuff) again about OpenLDAP thread-safety, = and he said that it is thread safe. > > Since when? Which version? Which lib? libldap_r is supposed to be = reentrant but that's not what python-ldap is built against. Furthermore = it was not possible to get a positive statement from Kurt Zeilenga last = time I asked. > > Ciao, Michael. |
|
From: <mi...@st...> - 2002-06-28 19:20:18
|
Joe Little wrote: > two snippets from Luke when speaking about 2.1 Note that most people still build against OpenLDAP 2.0.x. I don't want to mandate OpenLDAP 2.1.x at the moment because there will surely be more complaints about this than about bad performance. I don't like #ifdef's either. Lesson learned so far: I'm not a C programmer and the guys adding the #ifdef's won't be available for maintaining the C code later. > When I mentioned complaints here about thread safety, his response: > > I haven't seen those complaints. The OpenLDAP client library is guaranteed > to be thread safe as long as you (a) link against libldap_r and (b) do > not permit concurrent access to a single LDAP session (ie. use one per > thread or a mutex around it). Ok, not really new information. That was also my understanding from Kurt's postings quite a while ago. Caveats: 1. libldap_r has not been tested with python-ldap yet. 2. libldap_r is not fully implemented in OpenLDAP 2.0.x. AFAIK in 2.0 it's just a hack to make slurpd work. 3. We can't get completely rid of locking. We can do finer grained locking per LDAPObject instance. AFAIK global functions have still to be protected by module-wide locking. > So, libldap_r is one requirement... I'd be glad if someone dives into the gory details. > but the other seems to be already > set by our use of internal thread locking to that single session. At the moment there's a single module-wide lock serializing all calls into libldap since this is required with OpenLDAP 2.0.x. This can be easily changed off course. Ciao, Michael. |
|
From: Joe L. <jl...@op...> - 2002-06-28 19:27:39
|
On Friday, June 28, 2002, at 12:00 PM, Michael Str=F6der wrote: > Joe Little wrote: >> two snippets from Luke when speaking about 2.1 > > Note that most people still build against OpenLDAP 2.0.x. I don't want=20= > to mandate OpenLDAP 2.1.x at the moment because there will surely be=20= > more complaints about this than about bad performance. Agreed. I talked with Luke about 2.1's improvements in this area, and=20 he replied that 2.0 already was thread safe. I'll be diving into 2.1,=20 but 2.0 should be sufficient for coding thread safe python-ldap once it=20= uses libldap_r. > > I don't like #ifdef's either. Lesson learned so far: I'm not a C=20 > programmer and the guys adding the #ifdef's won't be available for=20 > maintaining the C code later. > >> When I mentioned complaints here about thread safety, his response: >> I haven't seen those complaints. The OpenLDAP client library is=20 >> guaranteed >> to be thread safe as long as you (a) link against libldap_r and (b) = do >> not permit concurrent access to a single LDAP session (ie. use one = per >> thread or a mutex around it). > > Ok, not really new information. That was also my understanding from=20 > Kurt's postings quite a while ago. > > Caveats: > 1. libldap_r has not been tested with python-ldap yet. > 2. libldap_r is not fully implemented in OpenLDAP 2.0.x. AFAIK in 2.0=20= > it's just a hack to make slurpd work. What I understood from Luke is that it has been completed in 2.x, with=20= bugs worked out along the way up to 2.0.25. > 3. We can't get completely rid of locking. We can do finer grained=20 > locking per LDAPObject instance. AFAIK global functions have still to=20= > be protected by module-wide locking. > >> So, libldap_r is one requirement... > > I'd be glad if someone dives into the gory details. > >> but the other seems to be already set by our use of internal thread=20= >> locking to that single session. > > At the moment there's a single module-wide lock serializing all calls=20= > into libldap since this is required with OpenLDAP 2.0.x. This can be=20= > easily changed off course. > Well, I hope this small discussion shows a way forward. > Ciao, Michael. > > > > ------------------------------------------------------- > This sf.net email is sponsored by:ThinkGeek > Caffeinated soap. No kidding. > http://thinkgeek.com/sf > _______________________________________________ > Python-LDAP-dev mailing list > Pyt...@li... > https://lists.sourceforge.net/lists/listinfo/python-ldap-dev |
|
From: <mi...@st...> - 2002-06-29 11:50:26
|
Joe Little wrote:
>=20
> On Friday, June 28, 2002, at 12:00 PM, Michael Str=F6der wrote:
>=20
>> Note that most people still build against OpenLDAP 2.0.x. I don't want=
=20
>> to mandate OpenLDAP 2.1.x at the moment because there will surely be=20
>> more complaints about this than about bad performance.
>=20
> Agreed. I talked with Luke about 2.1's improvements in this area, and h=
e=20
> replied that 2.0 already was thread safe. I'll be diving into 2.1, but =
> 2.0 should be sufficient for coding thread safe python-ldap once it use=
s=20
> libldap_r.
>=20
>> Caveats:
>> 1. libldap_r has not been tested with python-ldap yet.
>> 2. libldap_r is not fully implemented in OpenLDAP 2.0.x. AFAIK in 2.0 =
>> it's just a hack to make slurpd work.
>=20
> What I understood from Luke is that it has been completed in 2.x, with =
> bugs worked out along the way up to 2.0.25.
I tried to build python-ldap against libldap_r from OpenLDAP's=20
REL_ENG_2 branch by simply modifying setup.cfg:
libs =3D ldap_r lber sasl
The build went fine. ldd shows output like expected:
$ ldd build/lib.linux-i686-2.2/_ldap.so
libldap_r.so.2 =3D> /usr/lib/libldap_r.so.2 (0x4001f000)
liblber.so.2 =3D> /usr/lib/liblber.so.2 (0x4004b000)
libsasl.so.7 =3D> /usr/lib/libsasl.so.7 (0x40055000)
[..]
But import does not work.
$ python -c "import ldap"
Traceback (most recent call last):
File "<string>", line 1, in ?
File "/usr/lib/python2.2/site-packages/ldap/__init__.py", line=20
5, in ?
from _ldap import *
ImportError: /usr/lib/python2.2/site-packages/_ldap.so: undefined=20
symbol: ldap_first_reference
Anyone willing to dig into this?
Ciao, Michael.
|
|
From: <mi...@st...> - 2002-07-01 13:14:57
|
Michael Str=F6der wrote:
>=20
> I tried to build python-ldap against libldap_r from OpenLDAP's REL_ENG_=
2=20
> branch by simply modifying setup.cfg:
>=20
> libs =3D ldap_r lber sasl
>=20
> The build went fine. ldd shows output like expected:
> [..]
>=20
> But import does not work.
> [..]
> ImportError: /usr/lib/python2.2/site-packages/_ldap.so: undefined=20
> symbol: ldap_first_reference
Hmm I saw some postings on using libldap_r with other software=20
packages. They used libldap_r *and* libldap for linking.
I tried this in setup.cfg:
libs =3D ldap_r ldap lber sasl
It seems to work and ldd shows this:
$ ldd /usr/lib/python2.2/site-packages/_ldap.so
libldap_r.so.2 =3D> /usr/local/openldap2/lib/libldap_r.so.2=20
(0x4001f000)
libldap.so.2 =3D> /usr/local/openldap2/lib/libldap.so.2=20
(0x40052000)
liblber.so.2 =3D> /usr/local/openldap2/lib/liblber.so.2=20
(0x40082000)
libsasl.so.7 =3D> /usr/lib/libsasl.so.7 (0x4008e000)
Can anyone confirm that it's supposed to work that way using both=20
libldap_r and libldap for linking? How can I be sure that the=20
reentrant version of a ldaplib function is used? (Again revealing=20
my lack of C compiling knowledge...)
Ciao, Michael.
|
|
From: <mi...@st...> - 2002-07-01 14:04:01
|
Michael Str=F6der wrote:
>=20
> libs =3D ldap_r ldap lber sasl
libldap_r allows finer-grained locking for each LDAPObject=20
instance. I've checked in a new ldap/__init__.py and=20
ldap/ldapobject.py which allows to use separate Lock instances for=20
global function calls into LDAP lib and for each LDAPObject=20
instance created.
At the moment you have to create a LDAPObject instance directly=20
for using that (not through ldap.open() or ldap.initialize()).
Example:
l =3D ldap.ldapobject.LDAPObject('ldap://www.openldap.org',ldap_r=3D1)
Also there's a more elegant fall-back if the Python installation=20
used has no thread support (translates to import threading failed).
Please read the changed source and comment!
Please try to build a python-ldap with libldap_r and test!
Also test with Python without thread support!
Ciao, Michael.
|
|
From: Derrick 'd. H. <dm...@dm...> - 2002-06-28 22:59:17
|
On Fri, Jun 28, 2002 at 09:49:14AM -0700, Joe Little wrote:
| I haven't seen those complaints. The OpenLDAP client library is guaranteed
| to be thread safe as long as you
| do not permit concurrent access to a single LDAP session (ie. use
| one per thread or a mutex around it).
(FWIW)
This sounds perfectly acceptable to me.
Making a separate connection for each thread is likely to reduce
contention anyways (improves performance).
I don't see my patch (other thread) on the list, but don't use it --
it breaks web2ldap :-(. I need to work on it some more.
-D
--=20
The Consultant's Curse:
When the customer has beaten upon you long enough, give him
what he asks for, instead of what he needs. This is very strong
medicine, and is normally only required once.
=20
http://dman.ddts.net/~dman/
|
|
From: <mi...@st...> - 2002-06-29 12:01:22
|
Derrick 'dman' Hudson wrote: > On Fri, Jun 28, 2002 at 09:49:14AM -0700, Joe Little wrote: > > | do not permit concurrent access to a single LDAP session (ie. use > | one per thread or a mutex around it). > > (FWIW) > This sounds perfectly acceptable to me. > Making a separate connection for each thread is likely to reduce > contention anyways (improves performance). This remembers me of an old idea I had to implement persistent LDAP connection pooling with Queue.Queue found in Python's standard lib. E.g. if a HTTP server serves HTTP requests with 100 active threads you're not really keen to open a LDAP connection per thread because 1. your system ressources might get exhausted (e.g. file handles) 2. opening a connection is very slow (e.g. when initializing attributes read from RootDSE, SSL negotiation, SASL binds, etc.). Ciao, Michael. |
Thanks for helping keep SourceForge clean.
X