perl-ldap-dev

Large Searches

[David R. <d.r...@qu...>]

Hi,

I have scoured the archives and while found some interesting posts,
nothing that I could get to work for my situation.

I am performing a search that will return lots of entries from the
directory (about 50,000) and I am getting the error 'Out of memory!'
which is not suprising.  So, I started writing a search that uses a
callback, which will shift the returned entry off the results list, thus
(in theory) not running out of memory.

Here are the code segments:

printf( "Searching...\n" );
$count = 0;
$res = $ldap->search( base => $LDAP_BASE, filter =>
"(objectclass=QUTmailPerson)",
                      attrs => ['uid', 'mailUID', 'QUTmailbox',
'mailboxQuota', 'cn' ],
                      callback => &LDAP_callback() );

sub LDAP_callback {
my ( $mesg, $entry ) = @_;

	printf( "Start Callback\n");

	if( ! defined( $mesg ) ) {
		printf( "MESG Not defined\n" );
		if( ! defined( $entry ) ) {
        		printf( "ENT Not defined\n" );
		}
	} else {
      	  	$mesg->shift_entry();
		printf( "Got entry: %d\n", $count++ );
	}

	printf( "End Callback\n");
}

This is on the way to building a much larger peice of code, so for now
just knowing I got the entries will do.  So, this produces the output:

Searching...
Start Callback
MESG Not defined
ENT Not defined
End Callback
Can't use string ("1") as a subroutine ref while "strict refs" in use at
/usr/lib/perl-5.005/lib/site_perl/5.005/Net/LDAP/Search.pm line 51.


Any idea, help or pointers would be greatly appreciated.

Thanks,

-- 
David Richards
Project Manager (Messaging)
Information Technology Services
Queensland University of Technology

Re: Large Searches

[Graham B. <gb...@po...>]

On Thu, Mar 29, 2001 at 09:07:37AM +1000, David Richards wrote:
> printf( "Searching...\n" );
> $count = 0;
> $res = $ldap->search( base => $LDAP_BASE, filter =>
> "(objectclass=QUTmailPerson)",
>                       attrs => ['uid', 'mailUID', 'QUTmailbox',
> 'mailboxQuota', 'cn' ],
>                       callback => &LDAP_callback() );

That needs to be   \&LDAP_callback

In its current form you are calling the function and passing its results
to ->search() which is NOT what you want.

Graham.

Re: Large Searches

[David R. <d.r...@qu...>]

Graham,

Can you help me out here.  I am trying to retrieve a lot of entries that
causes the Perl-LDAP library to produce 'Out of memory!' and fall over. 
What is the best way to retrieve the entries?  Have you got some sample
code?

Thanks,

Dave.


Graham Barr wrote:
> 
> On Thu, Mar 29, 2001 at 09:07:37AM +1000, David Richards wrote:
> > printf( "Searching...\n" );
> > $count = 0;
> > $res = $ldap->search( base => $LDAP_BASE, filter =>
> > "(objectclass=QUTmailPerson)",
> >                       attrs => ['uid', 'mailUID', 'QUTmailbox',
> > 'mailboxQuota', 'cn' ],
> >                       callback => &LDAP_callback() );
> 
> That needs to be   \&LDAP_callback
> 
> In its current form you are calling the function and passing its results
> to ->search() which is NOT what you want.
> 
> Graham.

-- 
David Richards
Project Manager (Messaging)
Information Technology Services
Queensland University of Technology

Re: Large Searches

[Clif H. <cl...@di...>]

> 
> Graham,
> 
> Can you help me out here.  I am trying to retrieve a lot of entries that
> causes the Perl-LDAP library to produce 'Out of memory!' and fall over. 
> What is the best way to retrieve the entries?  Have you got some sample
> code?


A little more information may help.

What version of perl, perl-ldap, system OS.

How many entries are you trying to get: 1000, 10000, 100000.

Regards,

Clif






> 
> Thanks,
> 
> Dave.
> 
> 
> Graham Barr wrote:
> > 
> > On Thu, Mar 29, 2001 at 09:07:37AM +1000, David Richards wrote:
> > > printf( "Searching...\n" );
> > > $count = 0;
> > > $res = $ldap->search( base => $LDAP_BASE, filter =>
> > > "(objectclass=QUTmailPerson)",
> > >                       attrs => ['uid', 'mailUID', 'QUTmailbox',
> > > 'mailboxQuota', 'cn' ],
> > >                       callback => &LDAP_callback() );
> > 
> > That needs to be   \&LDAP_callback
> > 
> > In its current form you are calling the function and passing its results
> > to ->search() which is NOT what you want.
> > 
> > Graham.
> 
> -- 
> David Richards
> Project Manager (Messaging)
> Information Technology Services
> Queensland University of Technology
> 
> 

Re: Large Searches

[David R. <d.r...@qu...>]

OK:

perl -v produces:
This is perl, version 5.005_03 built for alpha-dec_osf

uname -a prduces:
OSF1 tu01m2.qut.edu.au V5.1 732 alpha

and perl-ldap is:
perl-ldap-0.22

I am attempting to retrieve about 120 000 entries, retrieving 5
attributes, which are (just for an idea of size):
	
	uid		: typically 8 characters
 	mailUID		: typically 89 characters
	QUTmailbox	: flag 'yes' or 'no'
	mailboxQuota	: Integer, most cases is 10000
	cn		: Normal common names.

The script is an account maintenance script for our central mail system,
to manage the active IMAP accounts, just for a bit of context.

Thanks,

Clif Harden wrote:
> 
> >
> > Graham,
> >
> > Can you help me out here.  I am trying to retrieve a lot of entries that
> > causes the Perl-LDAP library to produce 'Out of memory!' and fall over.
> > What is the best way to retrieve the entries?  Have you got some sample
> > code?
> 
> A little more information may help.
> 
> What version of perl, perl-ldap, system OS.
> 
> How many entries are you trying to get: 1000, 10000, 100000.
> 
> Regards,
> 
> Clif
> 
> >
> > Thanks,
> >
> > Dave.
> >
> >
> > Graham Barr wrote:
> > >
> > > On Thu, Mar 29, 2001 at 09:07:37AM +1000, David Richards wrote:
> > > > printf( "Searching...\n" );
> > > > $count = 0;
> > > > $res = $ldap->search( base => $LDAP_BASE, filter =>
> > > > "(objectclass=QUTmailPerson)",
> > > >                       attrs => ['uid', 'mailUID', 'QUTmailbox',
> > > > 'mailboxQuota', 'cn' ],
> > > >                       callback => &LDAP_callback() );
> > >
> > > That needs to be   \&LDAP_callback
> > >
> > > In its current form you are calling the function and passing its results
> > > to ->search() which is NOT what you want.
> > >
> > > Graham.
> >
> > --
> > David Richards
> > Project Manager (Messaging)
> > Information Technology Services
> > Queensland University of Technology
> >
> >

-- 
David Richards
Project Manager (Messaging)
Information Technology Services
Queensland University of Technology