python-ldap-dev

Why LDAPObject.._ldap_object_lock around library calls?

[Ken K. <key...@KS...>]

Hi Folks,

  I'm having a little trouble with the _ldap_object_lock in the
LDAPObject around ldap library function calls.  Is the link
to protect against threads making function calls when the _ldap
module is linked against the non-reentrant version of the libldap?
That is, wouldn't it be safe to get rid of the _ldap_object_lock
references in LDAPObject._ldap_call() if the module is linked
against libldap_r?  Or is there some other, non-threadsafe reason
that I'm missing?

Thanks for any info,
K^2

Re: Why LDAPObject.._ldap_object_lock around library calls?

[<mi...@st...>]

Ken Key wrote:
> 
>   I'm having a little trouble with the _ldap_object_lock in the
> LDAPObject around ldap library function calls.

Which trouble do you have?

>  Is the link
> to protect against threads making function calls when the _ldap
> module is linked against the non-reentrant version of the libldap?
> That is, wouldn't it be safe to get rid of the _ldap_object_lock
> references in LDAPObject._ldap_call() if the module is linked
> against libldap_r?  Or is there some other, non-threadsafe reason
> that I'm missing?

According to related postings on the OpenLDAP lists libldap_r is re-entrant 
on a per-connection basis. Therefore linking with libldap_r improves the 
situation since a finer-grained locking is used in LDAPObject class (see 
method _ldap_lock()).

Note that proper support for libldap_r also depends on the version of the 
OpenLDAP libs used.

Ciao, Michael.

Re: Why LDAPObject.._ldap_object_lock around library calls?

[Ken K. <key...@KS...>]

Ah, I found the answer to my question.  Since we store the thread
state in the LDAPObject instance, the object cannot be shared between
my two threads.

I tossed together a prototype of the two-thread model I was thinking of.
With the LDAPObject._ldap_lock in place, I got the deadlock I was
afraid of.  When I removed the _ldap_lock.acquire/release in
_ldap_call() I got the "saving thread twice?" fatal error from the
LDAP_BEGIN_ALLOW_THREADS() when the second thread makes the second
function call.

This message may arrive out-of-order with my original reply, sorry
about that.  I forgot to reset my From: line.

Cheers,
K^2

Re: Why LDAPObject.._ldap_object_lock around library calls?

[<mi...@st...>]

Ken Key wrote:
> Ah, I found the answer to my question.  Since we store the thread
> state in the LDAPObject instance, the object cannot be shared between
> my two threads.

It can be shared. That's what the locks are for. But not in a completely 
async manner like you're trying. Maybe your problem is that you are using 
result2() in a blocking manner. You can let it somewhat poll the results. 
But this is CPU intensive.

(History: I had a non-blocking version of result() in former versions of 
python-ldap to avoid locking over long time spans. But some people 
experienced serious problems with it.)

> I tossed together a prototype of the two-thread model I was thinking of.
> With the LDAPObject._ldap_lock in place, I got the deadlock I was
> afraid of.  When I removed the _ldap_lock.acquire/release in
> _ldap_call() I got the "saving thread twice?" fatal error from the
> LDAP_BEGIN_ALLOW_THREADS() when the second thread makes the second
> function call.

Don't mess around with the locks! They are needed because of the OpenLDAP libs.

Ciao, Michael.

Re: Why LDAPObject.._ldap_object_lock around library calls?

[<mi...@st...>]

Ken Key wrote:
> 
> I need to have two threads, a Producer and Consumer (relative to LDAP),
 > [..]
> translates the results into
> a form my proxy protocol and puts it on the work result Queue.

I probably don't understand what you're after. However I try to give some 
answers.

>>According to related postings on the OpenLDAP lists libldap_r is
>>re-entrant
>>on a per-connection basis. Therefore linking with libldap_r improves the
>>situation since a finer-grained locking is used in LDAPObject class (see
>>method _ldap_lock()).
> 
> However, the lock is still across all methods of an instance of the
> LDAPObject, and thus in force for all operations on the connection,
> across all threads.  That's the problem.

Your threads should use different LDAP connections thus LDAPObject instances.

> I was trying to determine if there were reasons beyond the
> underlying binary OpenLDAP client libraries that the locking was in
> place.

The OpenLDAP libs are the problem. Nothing in python-ldap requires the locks.

>  If not, I was thinking of overriding the _ldap_call() in
> my own class and eliminating it, since the ldap_r is a requirement
> for my proxy's environment.

Don't do this! The OpenLDAP libs are not re-entrant within one connection 
context. See OpenLDAP mailings list archives for details.

Ciao, Michael.

Re: Why LDAPObject.._ldap_object_lock around library calls?

[Ken K. <key...@KS...>]

> Don't do this! The OpenLDAP libs are not re-entrant within one connection
> context. See OpenLDAP mailings list archives for details.
>

Ahh.  This is the crux of my problem!  I was working on the mistaken
assumption that the ldap_r library verion was fully (operational)
thread-safe. As you can tell, I'm a newbie to LDAP as well as Python.

Thanks for saving me from myself!
K^2