python-ldap-dev

One result call per record

[Ron T. <Ro...@US...>]

Hi all,

We are experiencing something pretty odd.

CentOS 5  python2.4.4  python-ldap 2.3.4

While polling it appears that we have to call result for every record that
is waiting to be read.  Since we added a 250 millisecond delay between calls
larger queries are taking a really long time.

def searchForAsync(self, searchString):
   id = self.connection.search('dc=my,dc=base,dc=rec',2,searchStr
ing,['*','+'])
   result = [None, None]
   count = 0
   start = time.time()
   while(result[0] == None):
       count = count + 1
       result = self.connection.result(id, True, 0)
       time.sleep(0.25)
   	
   print "Total call count: " + str(count)
   print "Total query time: " + str(time.time - start)
   return result

The result: With 165 records the total call count is 167 and the total time
42.1 seconds.

Is this expected or did we do something wrong?

Thanks for your help!
Ron

Re: One result call per record

[Michael S. <mi...@st...>]

Ron Teitelbaum wrote:
> While polling it appears that we have to call result for every record that
> is waiting to be read.  Since we added a 250 millisecond delay between calls
> larger queries are taking a really long time.

Why do you wait such a long time in an extra time.sleep() call?

>        result = self.connection.result(id, True, 0)
>        time.sleep(0.25)

Hmm, I don't know very much of the inner workings of OpenLDAP's function 
ldap_result(). The behaviour also may depend on the version of OpenLDAP.

But how about fiddling around a little bit with the timeout argument for 
result()? You're blocking your while-loop with time.sleep() anyway. And 
the C wrapper module releases Python's GIL.

Something like

   [..within while loop..]
   try:
     result = l.result(msgid,True,0.25)
   except ldap.TIMEOUT:
     continue
   else:
     # Process result
   [..within while loop..]

I played around a little bit with the script attached on a local server 
with more than 1000 entries.

BTW: If you're after correctly dispatching results to several 
outstanding search requests you should probably use method result2() 
which also returns the message ID of the originating request:

http://python-ldap.sourceforge.net/doc/html/ldap.html#ldap.LDAPObject.result2

Use result3() if LDAPv3 extended controls are to be used to also receive 
controls sent by the server.

Ciao, Michael.

RE: One result call per record

[Ron T. <Ro...@US...>]

Hi Michael,

Thanks for your response.  I'll play around with it.  The sleep time added
was to enable us to test the behavior of result.  We are calling python
through Smalltalk and we add the sleep on our end (the calling thread in
Smalltalk not python) to enable other threads to process while waiting on
responses.  I guess I assumed that the result wouldn't take more then 1 or 2
waits, and the next call would retrieve the full result set.  I didn't
expect that I needed to call result once for each record.

Thanks again for your suggestions and all your help!

Ron

> -----Original Message-----
> From: Michael Ströder
> 
> Ron Teitelbaum wrote:
> > While polling it appears that we have to call result for every record
> that
> > is waiting to be read.  Since we added a 250 millisecond delay between
> calls
> > larger queries are taking a really long time.
> 
> Why do you wait such a long time in an extra time.sleep() call?
> 
> >        result = self.connection.result(id, True, 0)
> >        time.sleep(0.25)
> 
> Hmm, I don't know very much of the inner workings of OpenLDAP's function
> ldap_result(). The behaviour also may depend on the version of OpenLDAP.
> 
> But how about fiddling around a little bit with the timeout argument for
> result()? You're blocking your while-loop with time.sleep() anyway. And
> the C wrapper module releases Python's GIL.
> 
> Something like
> 
>    [..within while loop..]
>    try:
>      result = l.result(msgid,True,0.25)
>    except ldap.TIMEOUT:
>      continue
>    else:
>      # Process result
>    [..within while loop..]
> 
> I played around a little bit with the script attached on a local server
> with more than 1000 entries.
> 
> BTW: If you're after correctly dispatching results to several
> outstanding search requests you should probably use method result2()
> which also returns the message ID of the originating request:
> 
> http://python-
> ldap.sourceforge.net/doc/html/ldap.html#ldap.LDAPObject.result2
> 
> Use result3() if LDAPv3 extended controls are to be used to also receive
> controls sent by the server.
> 
> Ciao, Michael.