perl-ldap-dev

DSML library version .01

[Mark W. <mew...@un...>]

Hi,
Here's my first version of Net::DSML (and the helper module XML::DSML
which you could use seperately, but it's probably easier to use
Net::DSML).

For those who don't know about DSML, it's Directory Services Markup
Language which is the XML standard for exchanging directory services
data.

While it will likely replace LDIF as the primary mechanism to exchange
LDAP data, it is also designed to exchange other directory service data
like NIS and NDS. This is useful when you must load data between
disparate directory services asynchrously (sp).

Currently it will write and read only entry elements, schema support
will come in a future version (DSML schema is handled differently from
entries, I need to figure the best way to handle this & I may need to
make some changes/additions to Net::LDAP::Schema).

enjoy!

Mark

Re: DSML library version .01

[Graham B. <gb...@po...>]

On Thu, May 25, 2000 at 08:01:11PM -0500, Mark Wilcox wrote:
> Hi,
> Here's my first version of Net::DSML (and the helper module XML::DSML
> which you could use seperately, but it's probably easier to use
> Net::DSML).

This is a great start.

In XML::DSML you have a comment about renaming the module as Net::LDAP::DSML::Parser.

My answer would be that if it is a generic DSML parser, not connected with Net::LDAP,
then XML::DSML is probably the right place for it to live.

I have not tried the code, but from looking at it I have one concern.
It seems that the whole DSML file must be read into memory in one go.

This will be a disadvantage for large files and we should probably devise a way
to read the entries one at a time.

I guess you did it this way as by default XML::Parser works on the whole
file before it returns. But IIRC, it is possible to interface to
XML::Parser at a lower level. If not them maybe XML::Parser is not the
way to go.

Graham.

Re: DSML library version .01

[Mark W. <mew...@un...>]

Graham Barr wrote:

> On Thu, May 25, 2000 at 08:01:11PM -0500, Mark Wilcox wrote:
> > Hi,
> > Here's my first version of Net::DSML (and the helper module XML::DSML
> > which you could use seperately, but it's probably easier to use
> > Net::DSML).
>
> This is a great start.

Thanks. Last night I noticed a bug in my Base64 code for Net::LDAP::DSML, so I'm shipping
a new copy.

Also I've made a patch to Net::LDAP::Entry so that the dump() method will Base64 encocde
binary in the ouptut.

>
>
> In XML::DSML you have a comment about renaming the module as Net::LDAP::DSML::Parser.
>
> My answer would be that if it is a generic DSML parser, not connected with Net::LDAP,
> then XML::DSML is probably the right place for it to live.

Yeah that's what I'm thinking as well, but I wanted to get something out for people to
play with and then figure out the best way to go.

>
>
> I have not tried the code, but from looking at it I have one concern.
> It seems that the whole DSML file must be read into memory in one go.

I'll have to double-check this. I can't remember if XML::Parser's callback functions use
streaming or load the whole file.

>
>
> This will be a disadvantage for large files and we should probably devise a way
> to read the entries one at a time.

I'll double check, but worst-case I can rewrite it so that XML::Parser streams in the
file.

>

Mark

>

>
>
> I guess you did it this way as by default XML::Parser works on the whole
> file before it returns. But IIRC, it is possible to interface to
> XML::Parser at a lower level. If not them maybe XML::Parser is not the
> way to go.

>
>
> Graham.

Re: DSML library version .01

[Graham B. <gb...@po...>]

On Fri, May 26, 2000 at 08:58:20AM -0500, Mark Wilcox wrote:
> 
> 
> Graham Barr wrote:
> 
> > On Thu, May 25, 2000 at 08:01:11PM -0500, Mark Wilcox wrote:
> > > Hi,
> > > Here's my first version of Net::DSML (and the helper module XML::DSML
> > > which you could use seperately, but it's probably easier to use
> > > Net::DSML).
> >
> > This is a great start.
> 
> Thanks. Last night I noticed a bug in my Base64 code for Net::LDAP::DSML, so I'm shipping
> a new copy.
> 
> Also I've made a patch to Net::LDAP::Entry so that the dump() method will Base64 encocde
> binary in the ouptut.

OK, great.

> > In XML::DSML you have a comment about renaming the module as Net::LDAP::DSML::Parser.
> >
> > My answer would be that if it is a generic DSML parser, not connected with Net::LDAP,
> > then XML::DSML is probably the right place for it to live.
> 
> Yeah that's what I'm thinking as well, but I wanted to get something out for people to
> play with and then figure out the best way to go.
> 
> >
> >
> > I have not tried the code, but from looking at it I have one concern.
> > It seems that the whole DSML file must be read into memory in one go.
> 
> I'll have to double-check this. I can't remember if XML::Parser's callback functions use
> streaming or load the whole file.

XML::Parser I think always works on a whole file. But I think you can use XML::Parser::Expat
which is the low-level access.

> > This will be a disadvantage for large files and we should probably devise a way
> > to read the entries one at a time.
> 
> I'll double check, but worst-case I can rewrite it so that XML::Parser streams in the
> file.

Sure. One other thing to consider.

I assmue the DSML does not specify an order for things to appear in the file if
it contains entries and a schema. So you need to consider what to do if the
user asks to read an entry when the next thing is a schema, and visa-versa.

Graham.

Re: DSML library version .01

[Graham B. <gb...@po...>]

OK, I have been thinking about this some more.

On Fri, May 26, 2000 at 08:58:20AM -0500, Mark Wilcox wrote:
> > I have not tried the code, but from looking at it I have one concern.
> > It seems that the whole DSML file must be read into memory in one go.
> 
> I'll have to double-check this. I can't remember if XML::Parser's callback functions use
> streaming or load the whole file.

It's not that XML::Parser reads the whole file at one. It is that you read all
the entries from teh XML file into @results. So you are loading them all into
memory.

I have been considering if we really need an API like LDIF where the user does
read_entry() to get the next in the file.

From my reading of XML::Parser there is no way to do this. When you tell
it to parse it parses the whole file.

So I was thinking that maybe XML::DSML should just use callbacks itself.

For example have the user pass a sub reference which will be called
whenever an entry is read.

This can be extended when we start to read the schema, and we can have a different
sub to call. Or we can just have the user check the type of object passed.

I don't think this would be too much of a setback as I would think that most of
the time the current LDIF is used in a loop anyway.

So what we could end up with is something like

  $dsml = Net::LDAP::DSML->new();

  $dsml->process( $file, entry =>  \&process_entry, schema => \&process_schema )
     or die $dsml->error;

Now for writing we could do

  $dsml->open($file,"w");
  $dsml->write($entry);
  $dsml->write($schema);
  $dsml->close;

The open would write the header and the close would write the footer.

Thoughts ?

Graham.

Re: DSML library version .01

[Paul H. <hei...@cs...>]

On Fri, 26 May 2000, Graham Barr wrote:

> Now for writing we could do
> 
>   $dsml->open($file,"w");
>   $dsml->write($entry);
>   $dsml->write($schema);
>   $dsml->close;
> 
> The open would write the header and the close would write the footer.

If you passed something like an IO::File object to open(), you might gain
some flexibility if you wanted to write() to a pipe or something other
than a file.

Paul Heinlein
hei...@cs...

Re: DSML library version .01

[Graham B. <gb...@po...>]

On Fri, May 26, 2000 at 08:37:40AM -0700, Paul Heinlein wrote:
> On Fri, 26 May 2000, Graham Barr wrote:
> 
> > Now for writing we could do
> > 
> >   $dsml->open($file,"w");
> >   $dsml->write($entry);
> >   $dsml->write($schema);
> >   $dsml->close;
> > 
> > The open would write the header and the close would write the footer.
> 
> If you passed something like an IO::File object to open(), you might gain
> some flexibility if you wanted to write() to a pipe or something other
> than a file.

True, we can allow that. Also I don't think the "w" is really needed
as open would only be used for write.

Graham.

Re: DSML library version .01

[Mark W. <mew...@un...>]

Graham Barr wrote:

> OK, I have been thinking about this some more.
>
> On Fri, May 26, 2000 at 08:58:20AM -0500, Mark Wilcox wrote:
> > > I have not tried the code, but from looking at it I have one concern.
> > > It seems that the whole DSML file must be read into memory in one go.
> >
> > I'll have to double-check this. I can't remember if XML::Parser's callback functions use
> > streaming or load the whole file.
>
> It's not that XML::Parser reads the whole file at one. It is that you read all
> the entries from teh XML file into @results. So you are loading them all into
> memory.

Yeah, I took the quick and dirty route ;).

>
>
> I have been considering if we really need an API like LDIF where the user does
> read_entry() to get the next in the file.
>
> From my reading of XML::Parser there is no way to do this. When you tell
> it to parse it parses the whole file.
>
> So I was thinking that maybe XML::DSML should just use callbacks itself.
>
> For example have the user pass a sub reference which will be called
> whenever an entry is read.
>
> This can be extended when we start to read the schema, and we can have a different
> sub to call. Or we can just have the user check the type of object passed.
>
> I don't think this would be too much of a setback as I would think that most of
> the time the current LDIF is used in a loop anyway.
>
> So what we could end up with is something like
>
>   $dsml = Net::LDAP::DSML->new();
>
>   $dsml->process( $file, entry =>  \&process_entry, schema => \&process_schema )
>      or die $dsml->error;
>

Ok. I see. I'll start to work on it.

>
> Now for writing we could do
>
>   $dsml->open($file,"w");
>   $dsml->write($entry);
>   $dsml->write($schema);
>   $dsml->close;

Like this.

>
> The open would write the header and the close would write the footer.
>
> Thoughts ?

This is probably much better than what I had originally. I'll see what I get turned up.

Mark

>
>
> Graham.

Re: DSML library version .01

[Graham B. <gb...@po...>]

On Fri, May 26, 2000 at 10:46:51AM -0500, Mark Wilcox wrote:
> >   $dsml->process( $file, entry =>  \&process_entry, schema => \&process_schema )
> >      or die $dsml->error;
> >
> 
> Ok. I see. I'll start to work on it.

I have attached what I started on.

> 
> >
> > Now for writing we could do
> >
> >   $dsml->open($file,"w");
> >   $dsml->write($entry);
> >   $dsml->write($schema);
> >   $dsml->close;
> 
> Like this.

Good.

Graham.