perl-ldap-dev

Server "commit"

[Gordon S. <gor...@ce...>]

hello,

We're loading LDAP entries to iPlanet 4.1 Directory Server via Net::LDAP
v 0.22 using perl v5.6.0.

We can successfully read/modify/add/delete, a very good start.

However, we need to add on the order of 20k+ records. If all records are
attempted at the same time, the "journal" logs fill the hard drive after
about 400 additions. We can consistently add about 300 records, after
which we must exit the script and wait a few moments before running the
next batch. We wish to ultimately have a hands off one-shot process.

I've looked at the server configuration, perhaps to turn off
"journaling" temporarily, but have not seen anything promising yet.

Is there a way to "commit" the changes in progress from the perl script?
Having the script pause and/or unbind does not seem to do it.

   thanks,
   Gordon Smith

Re: Server "commit"

[Mark W. <mew...@un...>]

The answer is to get more hard-drive space or set the journals so that they
are written to a different partition that has more disk space (the latter is
also a good idea for disaster recover).

You cannot turn them off and you don't want to. Those files insure that your
directory will always be in a consistent state in case of a server failure.
For example if you lose power during an update. During the restart, the
server will rebuild itself, restoring the databases back to the last 'good'
commits.

That's a specific behavior to the Netscape DS, and is not something that is
part of LDAP. In LDAP there is no notion of commit and rollbacks.

Mark
Gordon Smith wrote:

> hello,
>
> We're loading LDAP entries to iPlanet 4.1 Directory Server via Net::LDAP
> v 0.22 using perl v5.6.0.
>
> We can successfully read/modify/add/delete, a very good start.
>
> However, we need to add on the order of 20k+ records. If all records are
> attempted at the same time, the "journal" logs fill the hard drive after
> about 400 additions. We can consistently add about 300 records, after
> which we must exit the script and wait a few moments before running the
> next batch. We wish to ultimately have a hands off one-shot process.
>
> I've looked at the server configuration, perhaps to turn off
> "journaling" temporarily, but have not seen anything promising yet.
>
> Is there a way to "commit" the changes in progress from the perl script?
> Having the script pause and/or unbind does not seem to do it.
>
>    thanks,
>    Gordon Smith

Re: Server "commit"

[Clif H. <cl...@di...>]

> 
> hello,
> 
> We're loading LDAP entries to iPlanet 4.1 Directory Server via Net::LDAP
> v 0.22 using perl v5.6.0.
> 
> We can successfully read/modify/add/delete, a very good start.
> 
> However, we need to add on the order of 20k+ records. If all records are
> attempted at the same time, the "journal" logs fill the hard drive after
> about 400 additions. We can consistently add about 300 records, after
> which we must exit the script and wait a few moments before running the
> next batch. We wish to ultimately have a hands off one-shot process.
> 
> I've looked at the server configuration, perhaps to turn off
> "journaling" temporarily, but have not seen anything promising yet.
> 
> Is there a way to "commit" the changes in progress from the perl script?
> Having the script pause and/or unbind does not seem to do it.
> 
>    thanks,
>    Gordon Smith
> 
> 

I have loaded 60K+ plus records on my Netscape 4.x directory server
using a script that does almost exactly what you describe.

The script binds, loads 250 records, unbinds, sleeps 2 minutes,
and then repeats this process until all records have completed. 
To the user this looks like one "long" continuos process.

There is a bulk loading script for Netscape servers that bypasses
ldap, you might want to look into using it.  I have never used it but
I have been told it is very fast and works quite well.  It comes with
the Netscape server software.

This is a problem that is not unique to Netscape, I have had this 
problem on 2 other directory servers also.  How sever the problem is
depends on computer speed, computer load, and amount of available
ram and disk space. 

Clif Harden           INTERNET:  c-h...@ti...