python-ldap-dev

Strange behavior in Python-LDAP 2.0.0pre04

[Mauro C. <mci...@si...>]

Hi all,
I'd wish to bring to your attention a bizarre thing that I noticed 
yesterday.

The documentation, at chapter 1.1.5.1 "LDAP operations", in the 
description of the result() method, states that

[...]
The result() method returns a tuple of the form |(/result-type/, 
/result-data/)|.
[...]

Now, I found that this isn't always true in the current (2.0.0pre04) 
implementation: in fact, when result() is called with the 'all' 
parameter set to non-zero (which BTW is the default), it returns just 
the /result-data/ part.

In other words, in this case the method returns just a list instead of a 
(string, list) tuple. This breaks all of my code that uses asynchronous 
methods, like for instance this snippet:

    l = ldap.open(LDAPServer)
    msgid = l.simple_bind(username, password)
    status,res = l.result(msgid, all=1, timeout=60)
    if status != 'RES_BIND':
        return -1

will always raise a UnpackTuple exception, since the return value of 
result() will be [] instead of ('RES_BIND', None) as I expected.

Someone may argue that the 'all' parameter is only significant in 
search() responses; this is true, but it still shouldn't cause 
undocumented behavior in the called function, IMHO.
Anyway, this snippet breaks too:

    msgid =
    l.search("o=myCompany.com",ldap.SCOPE_SUBTREE,'uid=a*',['cn','mail'])
    status,res = l.result(msgid, all=1, timeout=60)
    if status != 'RES_SEARCH_RESULT':
        return -1


I tried to figure out if something was wrong on my part, or if I was 
calling some other method in the wrong way, but it appears everything is 
correct and that in fact it's the result() implementation within 
ldapobject.py that does this: the underlying C calls are correct, and 
they return the right data structure.
The code in the result() implementation, if returning all data, simply 
appears to throw away the result type; it builds a list containing all 
results, and returns just that.

I'd like to point out again that calling result() with all=0 would 
return a tuple as documented, instead.
Moreover, the 1.x version of PythonLDAP result() worked 'correctly', 
i.e. exactly as documented in all cases.

I think it's strange that nobody noticed this so far, so I do think I'm 
doing something wrong, or that I missed the discussion on this list 
about changing the result() behavior.

May anyone shed light on this subject?

Thank you in advance

Mauro Cicognini

P.S. This happens because I've finally managed to compile python-ldap 
under native Windows 32 (no Cygwin) and I was doing a bit of testing. 
This was the only problem I noticed so far, but it doesn't look to be 
related to the C part at all...

Re: Strange behavior in Python-LDAP 2.0.0pre04

[<mi...@st...>]

Mauro Cicognini wrote:
> I'd wish to bring to your attention a bizarre thing that I noticed 
> yesterday.
> 
> The documentation, at chapter 1.1.5.1 "LDAP operations", in the 
> description of the result() method, states that
> 
> [...]
> The result() method returns a tuple of the form |(/result-type/, 
> /result-data/)|.
> [...]
> 
> Now, I found that this isn't always true in the current (2.0.0pre04) 
> implementation: in fact, when result() is called with the 'all' 
> parameter set to non-zero (which BTW is the default), it returns just 
> the /result-data/ part.

You're absolutely right.

> I'd like to point out again that calling result() with all=0 would 
> return a tuple as documented, instead.
> Moreover, the 1.x version of PythonLDAP result() worked 'correctly', 
> i.e. exactly as documented in all cases.
> 
> I think it's strange that nobody noticed this so far,

I noticed it and wanted to change it long ago but did not discuss 
that here. Thanks for reminding me about this open issue. I guess 
others didn't notice it since nobody calls result() with all=1. 
Instead others are using e.g. search_s() instead I guess.

I've implemented LDAPObject.result() in ldap.ldapobject which is 
mainly a wrapper around the internal result() method. The original 
result() is never called with all=1. Instead LDAPObject.result() 
handles that internally (also the timeout parameter). The goal was 
to turn all sync API calls into async API calls to avoid having to 
use the module-wide lock too long. (OpenLDAP 2 libs are not 
thread-safe.) The code is somewhat clunky and everybody is invited 
to dig into that.

Ciao, Michael.

Re: Strange behavior in Python-LDAP 2.0.0pre04

[Mauro C. <mci...@si...>]

Michael Ströder wrote:

> I noticed it and wanted to change it long ago but did not discuss that 
> here. Thanks for reminding me about this open issue. I guess others 
> didn't notice it since nobody calls result() with all=1. Instead 
> others are using e.g. search_s() instead I guess.

This is a welcome reassurance. I'm not going crazy then ;-)

> I've implemented LDAPObject.result() in ldap.ldapobject which is 
> mainly a wrapper around the internal result() method. The original 
> result() is never called with all=1. Instead LDAPObject.result() 
> handles that internally (also the timeout parameter). The goal was to 
> turn all sync API calls into async API calls to avoid having to use 
> the module-wide lock too long. (OpenLDAP 2 libs are not thread-safe.) 
> The code is somewhat clunky and everybody is invited to dig into that.

I have always used the async versions of the calls for the same reason. 
In fact I've never resorted to calling result() with all=0 so far, just 
because my result sets are (usually) on the smaller side.

So it's OK if I change the result() implementation within ldapobject.py?
My idea would be to make result() always return what's expected in the 
docos, i.e. always a tuple, emulating the 1.x behavior (and avoiding the 
breakage in my existing code).

Suggestions, anyone?

Mauro

Re: Strange behavior in Python-LDAP 2.0.0pre04

[<mi...@st...>]

Mauro Cicognini wrote:
>> The goal was to 
>> turn all sync API calls into async API calls to avoid having to use 
>> the module-wide lock too long. (OpenLDAP 2 libs are not thread-safe.) 
>> The code is somewhat clunky and everybody is invited to dig into that.
> 
> I have always used the async versions of the calls for the same reason. 
> In fact I've never resorted to calling result() with all=0 so far, just 
> because my result sets are (usually) on the smaller side.

Just use the search_s() or search_st() methods which are solely 
implemented in ldap.ldapobject.LDAPObject. I completely removed 
them from the C module part.

Set trace_level=1 when calling ldap.initialize() to track all the 
python-ldap API calls.

> So it's OK if I change the result() implementation within ldapobject.py?
> My idea would be to make result() always return what's expected in the 
> docos, i.e. always a tuple, emulating the 1.x behavior (and avoiding the 
> breakage in my existing code).

You're welcome to dig into that and find the bug I've made. But 
please keep in mind as requirement that the C implementation 
should never be called with all=1.

Note that ldap.ldapobject.LDAPObject.*_s() and 
ldap.ldapobject.LDAPObject.search_st() will also need some 
tweaking after fixing the bug in ldap.ldapobject.LDAPObject.result().

Ciao, Michael.

Win32 support (was: Strange behavior in Python-LDAP 2.0.0pre04)

[<mi...@st...>]

Mauro Cicognini wrote:
 > P.S. This happens because I've finally managed to compile
 > python-ldap under native Windows 32 (no Cygwin)

That's good news!

Can you package some Win32 binaries?
Maybe we can incorporate the necessary changes if any into 
python-ldap?

Ciao, Michael.

Re: Win32 support (was: Strange behavior in Python-LDAP 2.0.0pre04)

[Mauro C. <mci...@si...>]

Michael Ströder wrote:

> Mauro Cicognini wrote:
> > P.S. This happens because I've finally managed to compile
> > python-ldap under native Windows 32 (no Cygwin)
>
> That's good news!
>
> Can you package some Win32 binaries?
> Maybe we can incorporate the necessary changes if any into python-ldap?
>
I had to write a new setup.cfg; I don't know if one could just insert a 
win32 section in the existing one and make it work automagically. Maybe 
so, since setup.py is custom too.

That's not the end of the story, though. I need some help with 
distutils, and here's why.

On one hand, the actual LDAP (and LBER) libraries are made static; 
moreover, I was able to build them so that they link the correct shared 
objects (MSVCRT.DLL, i.e. the same that Python needs).
This way I could fold them into the _ldap shared object (_ldap.pyd) and 
not have bizarre runtime errors.

On the other hand, the SASL library gets built as a DLL; I can make it 
into a static library, although I don't really know if it would work; 
but the real problem is that the LDAP and LBER code looks for SASL 
support in a shared object.

I don't have time to delve into this and find out if OpenLDAP could be 
tweaked to support a statically linked SASL library, and, besides, the 
build procedure is already complex enough (and officially experimental: 
your mileage may vary).

So I just put my LIBSASL.DLL into the Windows system directory; it works :-)

However, to replicate this arrangement on other machines I'd have to 
tell distutils that a) the binary distribution needs to include 
LIBSASL.DLL, too; and b) when installing it, it needs to go somewhere on 
the PATH. I have no idea how.

Any suggestions?

Thanks,
Mauro

Re: Win32 support (was: Strange behavior in Python-LDAP 2.0.0pre04)

[<mi...@st...>]

Mauro Cicognini wrote:
 >
> That's not the end of the story, though. I need some help with 
> distutils, and here's why.
 > [..]
> However, to replicate this arrangement on other machines I'd have to 
> tell distutils that a) the binary distribution needs to include 
> LIBSASL.DLL, too; and b) when installing it, it needs to go somewhere on 
> the PATH. I have no idea how.

I'd recommend to just make this a installation requirement. 
Exactly like the OpenLDAP and SASL libs have to be in place in a 
Unix environment *before* you build python-ldap.

Ciao, Michael.