perl-ldap-dev

[Fwd] Problem with Net::LDAP

[Graham B. <gb...@po...>]

----- Forwarded message from Simon Allard <sim...@st...> -----

Date: Tue, 29 May 2001 17:51:59 +1200 (NZST)
To: <gb...@po...>
From: Simon Allard <sim...@st...>
Subject: Problem with Net::LDAP

Hey Graham.

I am hopeing you can shed some light on this. I have this problem where
your module seems to chew up a lot of memory.

I am running openldap 2.0.7 with perl-ldap-0.23.

In openldap I have Aprox 95000 entrys. Each entry looks similar to this:

dn:loginName=dbs, ou=auth, dc=domain, dc=co, dc=nz
objectclass: AuthAccount
loginName: dbs
userPassword: {crypt}sgaCZydxWDFD
nasAccess: SOMETHING
smtpAuth: 0
overrideProfile: 1
multicast: 0

When I do a basic search eg:

if (!$ldap->bind("cn=Manager, $authbase", password => $authpass, async => 1)) {
	print "Ldap Bind failed!\n";
	exit 1;
}
my $mesg = $ldap->search(
        base => "$authbase",
        filter => '(objectclass=ihugAuthAccount)'
       );

$ldap->unbind;

This search seems to chew up over 250meg of Ram. It pretty much makes the
perl module unusable. Do you have any suggestions on how I could get the
memory usage down to something reasonable?

Also doing the unbind seems to take a very long time :(

Hope you can help.
Thanks in Advance.

Regards
Simon Allard



Simon Allard (Senior Tool Monkey)
IHUG
Ph (09) 358-5067   Email: sim...@st...

The real fighting men and women of the 21st century will continue
to be techo's backed up by the grunts with bad haircuts and rifles.


----- End forwarded message -----

Re: [Fwd] Problem with Net::LDAP

[Graham B. <gb...@po...>]

It is taking so much memory because it is reading all the results into
memory before returning for you to process them.

I suspect you don't really need all of the entries, so try modifying your
filter to result the resutl set that is being sent.

If this is not possible, try using the callback option. This allows you
to process each entry as it arrives from the server.

Graham.

On Tue, May 29, 2001 at 09:41:54AM +0100, Graham Barr wrote:
> ----- Forwarded message from Simon Allard <sim...@st...> -----
> 
> Date: Tue, 29 May 2001 17:51:59 +1200 (NZST)
> To: <gb...@po...>
> From: Simon Allard <sim...@st...>
> Subject: Problem with Net::LDAP
> 
> Hey Graham.
> 
> I am hopeing you can shed some light on this. I have this problem where
> your module seems to chew up a lot of memory.
> 
> I am running openldap 2.0.7 with perl-ldap-0.23.
> 
> In openldap I have Aprox 95000 entrys. Each entry looks similar to this:
> 
> dn:loginName=dbs, ou=auth, dc=domain, dc=co, dc=nz
> objectclass: AuthAccount
> loginName: dbs
> userPassword: {crypt}sgaCZydxWDFD
> nasAccess: SOMETHING
> smtpAuth: 0
> overrideProfile: 1
> multicast: 0
> 
> When I do a basic search eg:
> 
> if (!$ldap->bind("cn=Manager, $authbase", password => $authpass, async => 1)) {
> 	print "Ldap Bind failed!\n";
> 	exit 1;
> }
> my $mesg = $ldap->search(
>         base => "$authbase",
>         filter => '(objectclass=ihugAuthAccount)'
>        );
> 
> $ldap->unbind;
> 
> This search seems to chew up over 250meg of Ram. It pretty much makes the
> perl module unusable. Do you have any suggestions on how I could get the
> memory usage down to something reasonable?
> 
> Also doing the unbind seems to take a very long time :(
> 
> Hope you can help.
> Thanks in Advance.
> 
> Regards
> Simon Allard
> 
> 
> 
> Simon Allard (Senior Tool Monkey)
> IHUG
> Ph (09) 358-5067   Email: sim...@st...
> 
> The real fighting men and women of the 21st century will continue
> to be techo's backed up by the grunts with bad haircuts and rifles.
> 
> 
> ----- End forwarded message -----

Re: [Fwd] Problem with Net::LDAP

[Chris R. <chr...@me...>]

Graham Barr <gb...@po...> wrote:
> It is taking so much memory because it is reading all the results into
> memory before returning for you to process them.
> 
> I suspect you don't really need all of the entries, so try modifying your
> filter to result the resutl set that is being sent.
> 
> If this is not possible, try using the callback option. This allows you
> to process each entry as it arrives from the server.
> 
> Graham.

This is true, however 95000 * 200 bytes (rough size of the example entry)
only comes to about 18 megabytes, and Simon said his perl process was using
250 megabytes :-(

That's a lot of overhead, unless Simon missed a zero out of his entry count
- 950000 entries would work out to use about the observed quantity of
memory. Well, the same order of magnitude anyway.

Cheers,

Chris

Re: [Fwd] Problem with Net::LDAP

[Clif H. <cl...@di...>]

> 
> Graham Barr <gb...@po...> wrote:
> > It is taking so much memory because it is reading all the results into
> > memory before returning for you to process them.
> > 
> > I suspect you don't really need all of the entries, so try modifying your
> > filter to result the resutl set that is being sent.
> > 
> > If this is not possible, try using the callback option. This allows you
> > to process each entry as it arrives from the server.
> > 
> > Graham.
> 
> This is true, however 95000 * 200 bytes (rough size of the example entry)
> only comes to about 18 megabytes, and Simon said his perl process was using
> 250 megabytes :-(
> 
> That's a lot of overhead, unless Simon missed a zero out of his entry count
> - 950000 entries would work out to use about the observed quantity of
> memory. Well, the same order of magnitude anyway.
> 
> Cheers,
> 
> Chris
> 
> 


It would be nice to know what version of perl he is running
and on what type of systems.

Regards,

Clif 

Re: [Fwd] Problem with Net::LDAP

[Simon A. <sim...@st...>]

> > This is true, however 95000 * 200 bytes (rough size of the example entry)
> > only comes to about 18 megabytes, and Simon said his perl process was using
> > 250 megabytes :-(
> >
> > That's a lot of overhead, unless Simon missed a zero out of his entry count
> > - 950000 entries would work out to use about the observed quantity of
> > memory. Well, the same order of magnitude anyway.
>
> It would be nice to know what version of perl he is running
> and on what type of systems.

Output from perl -v
This is perl, version 5.003 with EMBED
        built under solaris at Jun 10 1998 02:26:14
        + suidperl security patch

The perl is the version that comes with Oracle 9i.

delphi:~$ uname -a
SunOS delphi.ihug.co.nz 5.6 Generic_105181-20 sun4u sparc SUNW,Ultra-2




Simon Allard (Senior Tool Monkey)
IHUG
Ph (09) 358-5067   Email: sim...@st...

The real fighting men and women of the 21st century will continue
to be techo's backed up by the grunts with bad haircuts and rifles.

Re: [Fwd] Problem with Net::LDAP

[Clif H. <ch...@po...>]

Simon Allard wrote:
> 
> > > This is true, however 95000 * 200 bytes (rough size of the example entry)
> > > only comes to about 18 megabytes, and Simon said his perl process was using
> > > 250 megabytes :-(
> > >
> > > That's a lot of overhead, unless Simon missed a zero out of his entry count
> > > - 950000 entries would work out to use about the observed quantity of
> > > memory. Well, the same order of magnitude anyway.
> >
> > It would be nice to know what version of perl he is running
> > and on what type of systems.
> 
> Output from perl -v
> This is perl, version 5.003 with EMBED
>         built under solaris at Jun 10 1998 02:26:14
>         + suidperl security patch
> 
> The perl is the version that comes with Oracle 9i.
> 
> delphi:~$ uname -a
> SunOS delphi.ihug.co.nz 5.6 Generic_105181-20 sun4u sparc SUNW,Ultra-2
> 
> Simon Allard (Senior Tool Monkey)
> IHUG
> Ph (09) 358-5067   Email: sim...@st...
> 
> The real fighting men and women of the 21st century will continue
> to be techo's backed up by the grunts with bad haircuts and rifles.


Simon,

The lowest version of perl that we support is version 5.004.
Very soon we will only support 5.005_03 and above.

IMO your problem is in your version of perl.
 
Regards,

Clif Harden          ch...@po...

Re: [Fwd] Problem with Net::LDAP

[Simon A. <sim...@st...>]

> > Output from perl -v
> > This is perl, version 5.003 with EMBED
> >         built under solaris at Jun 10 1998 02:26:14
> >         + suidperl security patch
> >
> > The perl is the version that comes with Oracle 9i.
> >
> > delphi:~$ uname -a
> > SunOS delphi.ihug.co.nz 5.6 Generic_105181-20 sun4u sparc SUNW,Ultra-2
> Simon,
>
> The lowest version of perl that we support is version 5.004.
> Very soon we will only support 5.005_03 and above.
>
> IMO your problem is in your version of perl.

I have just tried it on my debian box.

soundy@zippie:~$ uname -a
Linux zippie 2.2.18 #2 Thu Apr 5 00:56:03 NZST 2001 i686 unknown


soundy@zippie:~$ perl -v
This is perl, v5.6.0 built for i386-linux
Copyright 1987-2000, Larry Wall

Source:
# Connect and bind to the LDAP Database
my $ldap = Net::LDAP->new('delphi.ihug.co.nz') or do {
        print "Couldn't connect to LDAP database $@\n";
        exit 1;
};

if (!$ldap->bind("cn=Manager, $authbase", password => $authpass)) {
        print "Ldap Bind failed!\n";
        exit 1;
}

my $mesg = $ldap->search(
        base => "$authbase",
        filter => '(objectclass=ihugAuthAccount)',


It used the same amount of Ram.


Simon Allard (Senior Tool Monkey)
IHUG
Ph (09) 358-5067   Email: sim...@st...

The real fighting men and women of the 21st century will continue
to be techo's backed up by the grunts with bad haircuts and rifles.

UNSUBSCRIBE

[christophe O. <chr...@ce...>]

-----Message d'origine-----
De : per...@li...
[mailto:per...@li...]De la part de Clif
Harden
Envoy=E9 : mercredi 30 mai 2001 03:46
=C0 : Simon Allard
Cc : LDAP Mailing List
Objet : Re: [Fwd] Problem with Net::LDAP


Simon Allard wrote:
>
> > > This is true, however 95000 * 200 bytes (rough size of the example
entry)
> > > only comes to about 18 megabytes, and Simon said his perl process w=
as
using
> > > 250 megabytes :-(
> > >
> > > That's a lot of overhead, unless Simon missed a zero out of his ent=
ry
count
> > > - 950000 entries would work out to use about the observed quantity =
of
> > > memory. Well, the same order of magnitude anyway.
> >
> > It would be nice to know what version of perl he is running
> > and on what type of systems.
>
> Output from perl -v
> This is perl, version 5.003 with EMBED
>         built under solaris at Jun 10 1998 02:26:14
>         + suidperl security patch
>
> The perl is the version that comes with Oracle 9i.
>
> delphi:~$ uname -a
> SunOS delphi.ihug.co.nz 5.6 Generic_105181-20 sun4u sparc SUNW,Ultra-2
>
> Simon Allard (Senior Tool Monkey)
> IHUG
> Ph (09) 358-5067   Email: sim...@st...
>
> The real fighting men and women of the 21st century will continue
> to be techo's backed up by the grunts with bad haircuts and rifles.


Simon,

The lowest version of perl that we support is version 5.004.
Very soon we will only support 5.005_03 and above.

IMO your problem is in your version of perl.

Regards,

Clif Harden          ch...@po...

Re: [Fwd] Problem with Net::LDAP

[Simon A. <sim...@st...>]

> This is true, however 95000 * 200 bytes (rough size of the example entry)
> only comes to about 18 megabytes, and Simon said his perl process was using
> 250 megabytes :-(
>
> That's a lot of overhead, unless Simon missed a zero out of his entry count
> - 950000 entries would work out to use about the observed quantity of
> memory. Well, the same order of magnitude anyway.

Unfortantly its is only 95000 entrys. I can read exactly the same amount
of data out of our Oracle databae and use ony 20meg of RAM.


Simon Allard (Senior Tool Monkey)
IHUG
Ph (09) 358-5067   Email: sim...@st...

The real fighting men and women of the 21st century will continue
to be techo's backed up by the grunts with bad haircuts and rifles.

Re: [Fwd] Problem with Net::LDAP

[Simon A. <sim...@st...>]

Thanks for the Reply Graham.

Callbacks sound like a great idea, but I am not having much luck with
them. There isn't much documentation about them. I have tried in on my
Solaid box with perl 5.003 and my Debian box  with perl 5.6 and get the
same result.

Source follows:

#!/usr/bin/perl

use strict;
use Net::LDAP;

my $authbase = "ou=auth, dc=ihug, dc=co, dc=nz";
my $authpass = "secret";


# Connect and bind to the LDAP Database
my $ldap = Net::LDAP->new('delphi.ihug.co.nz') or do {
	print "Couldn't connect to LDAP database $@\n";
	exit 1;
};

if (!$ldap->bind("cn=Manager, $authbase", password => $authpass)) {
	print "Ldap Bind failed!\n";
	exit 1;
}

$ldap->search(
        base => "$authbase",
        filter => '(objectclass=ihugAuthAccount)',
	callback => &check_oracle()
);


sub check_oracle {
        my ($mesg, $entry) = @_;

        $entry->dump();
}


When I run this I get:
delphi:~$ ./ldap-test.pl
Can't call method "dump" on an undefined value at ./ldap-test.pl line 31.

It looks like its calling the right sub, but its not getting passed the
varables? Any ideas why?

Thanks in Advance

- Simon Allard



> If this is not possible, try using the callback option. This allows you
> to process each entry as it arrives from the server.
>
> Graham.
>
> On Tue, May 29, 2001 at 09:41:54AM +0100, Graham Barr wrote:
> > ----- Forwarded message from Simon Allard <sim...@st...> -----
> >
> > Date: Tue, 29 May 2001 17:51:59 +1200 (NZST)
> > To: <gb...@po...>
> > From: Simon Allard <sim...@st...>
> > Subject: Problem with Net::LDAP
> >
> > Hey Graham.
> >
> > I am hopeing you can shed some light on this. I have this problem where
> > your module seems to chew up a lot of memory.
> >
> > I am running openldap 2.0.7 with perl-ldap-0.23.
> >
> > In openldap I have Aprox 95000 entrys. Each entry looks similar to this:
> >
> > dn:loginName=dbs, ou=auth, dc=domain, dc=co, dc=nz
> > objectclass: AuthAccount
> > loginName: dbs
> > userPassword: {crypt}sgaCZydxWDFD
> > nasAccess: SOMETHING
> > smtpAuth: 0
> > overrideProfile: 1
> > multicast: 0
> >
> > When I do a basic search eg:
> >
> > if (!$ldap->bind("cn=Manager, $authbase", password => $authpass, async => 1)) {
> > 	print "Ldap Bind failed!\n";
> > 	exit 1;
> > }
> > my $mesg = $ldap->search(
> >         base => "$authbase",
> >         filter => '(objectclass=ihugAuthAccount)'
> >        );
> >
> > $ldap->unbind;
> >
> > This search seems to chew up over 250meg of Ram. It pretty much makes the
> > perl module unusable. Do you have any suggestions on how I could get the
> > memory usage down to something reasonable?
> >
> > Also doing the unbind seems to take a very long time :(
> >
> > Hope you can help.
> > Thanks in Advance.
> >
> > Regards
> > Simon Allard
> >
> >
> >
> > Simon Allard (Senior Tool Monkey)
> > IHUG
> > Ph (09) 358-5067   Email: sim...@st...
> >
> > The real fighting men and women of the 21st century will continue
> > to be techo's backed up by the grunts with bad haircuts and rifles.
> >
> >
> > ----- End forwarded message -----
>

Simon Allard (Senior Tool Monkey)
IHUG
Ph (09) 358-5067   Email: sim...@st...

The real fighting men and women of the 21st century will continue
to be techo's backed up by the grunts with bad haircuts and rifles.

Re: [Fwd] Problem with Net::LDAP

[Graham B. <gb...@po...>]

On Thu, May 31, 2001 at 03:01:26PM +1200, Simon Allard wrote:
> Thanks for the Reply Graham.
> 
> Callbacks sound like a great idea, but I am not having much luck with
> them. There isn't much documentation about them. I have tried in on my
> Solaid box with perl 5.003 and my Debian box  with perl 5.6 and get the
> same result.
> 
> Source follows:

> $ldap->search(
>         base => "$authbase",
>         filter => '(objectclass=ihugAuthAccount)',
> 	callback => &check_oracle()
> );

You need to pass a reference to the sub. This calls the sub and passes
its result.

  callback => \&check_oracle


> sub check_oracle {
>         my ($mesg, $entry) = @_;
> 
>         $entry->dump();
> }

Entry will not be defined on the last call, so make sure you check it is defined.
Also to save memoery you need to tell Net::LDAP to remove it from the list.

 sub check_oracle {
         my ($mesg, $entry) = @_;
 
         if ($entry and $entry->isa('Net::LDAP::Entry')) {
              $entry->dump();
              $mesg->pop_entry;
         }
 }

Graham.

Re: [Fwd] Problem with Net::LDAP

[Clif H. <cl...@di...>]

> 
> Entry will not be defined on the last call, so make sure you check it is defined.
> Also to save memoery you need to tell Net::LDAP to remove it from the list.
> 
>  sub check_oracle {
>          my ($mesg, $entry) = @_;
>  
>          if ($entry and $entry->isa('Net::LDAP::Entry')) {
>               $entry->dump();
>               $mesg->pop_entry;
>          }
>  }
> 
> Graham.
> 
> 

Graham,

We need to document entry->dump method. 
I do not see it listed in the online documentation.

The mesg->pop_entry method probably needs to be listed in the 
Message documentation too, I could currently only find it in the
Seach documentation.

This info will probably be put into the FAQ in a new callback section.

Regards,

Clif Harden

Re: [Fwd] Problem with Net::LDAP

[Graham B. <gb...@po...>]

On Thu, May 31, 2001 at 08:12:39AM -0500, Clif Harden wrote:
> > 
> > Entry will not be defined on the last call, so make sure you check it is defined.
> > Also to save memoery you need to tell Net::LDAP to remove it from the list.
> > 
> >  sub check_oracle {
> >          my ($mesg, $entry) = @_;
> >  
> >          if ($entry and $entry->isa('Net::LDAP::Entry')) {
> >               $entry->dump();
> >               $mesg->pop_entry;
> >          }
> >  }
> > 
> > Graham.
> > 
> > 
> 
> Graham,
> 
> We need to document entry->dump method. 
> I do not see it listed in the online documentation.

It needs to be fixed first to use Net::LDAP::LDIF

> The mesg->pop_entry method probably needs to be listed in the 
> Message documentation too, I could currently only find it in the
> Seach documentation.

No, Net::LDAP::Search is the right place. It is not a method in
Net::LDAP::Message. Net::LDAP::Search is a sub-class of Net::LDAP::Message
that is returned by ->search. It is here that the pop_entry method
is defined.

Graham.

> 
> This info will probably be put into the FAQ in a new callback section.
> 
> Regards,
> 
> Clif Harden
> 

Re: [Fwd] Problem with Net::LDAP

[Christoph N. <en...@ap...>]

On Thu, 31 May 2001, Graham Barr wrote:

> On Thu, May 31, 2001 at 08:12:39AM -0500, Clif Harden wrote:
> > > 
> > > Entry will not be defined on the last call, so make sure you check it is defined.
> > > Also to save memoery you need to tell Net::LDAP to remove it from the list.
> > > 
> > >  sub check_oracle {
> > >          my ($mesg, $entry) = @_;
> > >  
> > >          if ($entry and $entry->isa('Net::LDAP::Entry')) {
> > >               $entry->dump();
> > >               $mesg->pop_entry;
> > >          }
> > >  }
> > > 
> > > Graham.
> > > 
> > > 
> > 
> > Graham,
> > 
> > We need to document entry->dump method. 
> > I do not see it listed in the online documentation.
> 
> It needs to be fixed first to use Net::LDAP::LDIF
> 
> > The mesg->pop_entry method probably needs to be listed in the 
> > Message documentation too, I could currently only find it in the
> > Seach documentation.
> 
> No, Net::LDAP::Search is the right place. It is not a method in
> Net::LDAP::Message. Net::LDAP::Search is a sub-class of Net::LDAP::Message
> that is returned by ->search. It is here that the pop_entry method
> is defined.
> 
> Graham.
> 
> > 
> > This info will probably be put into the FAQ in a new callback section.
> > 

I think an explaination of how to create a callback that removes each
entry as it comes should definitely be put in the FAQ.  I recall having to
read through the source code in order to figure this out before.  Knowing
how to do this is indespensible if one needs to process large numbers of
entries from LDAP.

- Christoph


> > Regards,
> > 
> > Clif Harden
> > 
> 
> 

Re: [Fwd] Problem with Net::LDAP

[Simon A. <sim...@st...>]

> Entry will not be defined on the last call, so make sure you check it is defined.
> Also to save memoery you need to tell Net::LDAP to remove it from the list.
>
>  sub check_oracle {
>          my ($mesg, $entry) = @_;
>
>          if ($entry and $entry->isa('Net::LDAP::Entry')) {
>               $entry->dump();
>               $mesg->pop_entry;
>          }
>  }
>
> Graham.

Thanks heaps for that Graham. It works wonders :-)


Simon Allard (Senior Tool Monkey)
IHUG
Ph (09) 358-5067   Email: sim...@st...

The real fighting men and women of the 21st century will continue
to be techo's backed up by the grunts with bad haircuts and rifles.