plob-discussion

[PLOB] Upcoming release 2.11

[Heiko K. <hei...@gm...>]

All,

I'm preparing release 2.11 of Plob. The new release will
include a more stable version of Plob's server code (i.e.,
persistent heap) for Windows; furthermore, I'm updating
Plob so that it will run on the actual LISP versions of
LispWorks (4.4.x) and Franz' ACL (7.x). I'm still looking
for an Irix account for checking Plob there, so please
contact me if someone is willing to give me an Irix
account.

I have still plans to replace PLOB's underlying library
POSTORE with Berkely DB, but unfortunately did
not have the time to do so.

Please get in contact with me if you have unresolved
issues, if possible, I will try to solve these in the upcoming
release.

I would like to thank Franz Inc. and LispWorks for providing
me with eval licenses for my development work on Plob.

Viele Grüße,
Heiko

Heiko Kirschke
Achtern Barg 48
D-22885 Barsbüttel
Tel.: +49 40 225497

Re: [PLOB] Upcoming release 2.11

[Edi W. <ed...@ag...>]

Hi!

On Wed, 06 Apr 2005 08:52:13 +0200, Heiko Kirschke <hei...@gm...> wrote:

> I'm preparing release 2.11 of Plob. The new release will include a
> more stable version of Plob's server code (i.e., persistent heap)
> for Windows; furthermore, I'm updating Plob so that it will run on
> the actual LISP versions of LispWorks (4.4.x) and Franz' ACL (7.x).

Cool, thanks.

> I have still plans to replace PLOB's underlying library POSTORE with
> Berkely DB, but unfortunately did not have the time to do so.

It'd be great if you could replace the underlying store with something
else.  I wonder, however, if Berkeley DB is the right solution.
Again, because of licensing (or rather pricing) issues.  See my email
to the Elephant mailing list:

  <http://common-lisp.net/pipermail/elephant-devel/2005-March/000121.html>

Could you describe in a few words which features have to be supported
by an alternative back-end?

Thanks for all on your work on this.

Greetings from Hamburg,
Edi.

Re: [PLOB] Upcoming release 2.11

[Heiko K. <hei...@gm...>]

Moin Edi,

Edi Weitz schrieb:

>>I have still plans to replace PLOB's underlying library POSTORE with
>>Berkely DB, but unfortunately did not have the time to do so.
>>    
>>
>It'd be great if you could replace the underlying store with something
>else.  I wonder, however, if Berkeley DB is the right solution.
>Again, because of licensing (or rather pricing) issues.  See my email
>to the Elephant mailing list:
>
>  <http://common-lisp.net/pipermail/elephant-devel/2005-March/000121.html>
>  
>
Ok, reading this means that the Berkeley DB license is not suitable for 
Plob. Sigh.

>Could you describe in a few words which features have to be supported
>by an alternative back-end?
>
The requirements are not that high.

First, I know of a successor of POSTORE called `Lumberjack' with the 
same API. Unfortunately, it looks as if the work on Lumberjack has stuck 
a bit, since the author(s) told me that it is not yet ready for putting 
it into the public domain. I will ask them again.

Second, to me it looks as if  Plob should use its own underlying 
persistent heap. There are several options to this:

1. Abonding the mixed C-/LISP-approach of Plob, coding the persistent 
heap in LISP. Would mean a major redesign and cost a lot of effort.

2. Replace the POSTORE lib by an own implementation of a persistent 
heap, again in C. Would mean only a redesign on Plob's lower levels, 
with maybe moderate efforts. I cannot estimate what it would mean to 
replace the POSTORE lib by something coded in LISP, but I'm afraid that 
this would mean that I've to drop the idea of using Plob randomly in 
server or serverless mode and to decide on supporting only exactly one 
of these modes.

For option 2., the POSTORE API is not that complicated. There are 
functions for allocating a persistent record with a quite simple 
structure (references to other persistent  records and values [i.e., no 
references]), access functions to these records by their `object id', 
and some administrative functions like flushing, creating a new 
persistent heap etc. You find the interface in the SH_* functions 
declared in plob-2.10/src/include/postore.h. All other databasespecific 
stuff (transactions, btrees etc.) has been implemented in Plob anyway in 
the layer above the persistent heap.

Since my time for working on Plob is rather constrained, I would prefer 
option 2.

Viele Grüße from Hamburg, too,
Heiko

Heiko Kirschke
Achtern Barg 48
D-22885 Barsbüttel
Tel.: +49 40 225497

Re: [PLOB] Upcoming release 2.11

[Edi W. <ed...@ag...>]

Moin!

On Wed, 06 Apr 2005 10:03:57 +0200, Heiko Kirschke <hei...@gm...> wrote:

> Second, to me it looks as if Plob should use its own underlying
> persistent heap. There are several options to this:
>
> 1. Abonding the mixed C-/LISP-approach of Plob, coding the
> persistent heap in LISP. Would mean a major redesign and cost a lot
> of effort.

But I seriously wonder if it's not worth it.  A pure CL solution will
definitely be easier to debug and port, and it'll be easier to deliver
applications based on it.  Plus, as Plob is an open source project, we
should not forget that it's most likely much more fun for Lisp hackers
to work on Lisp code than to work on C code... :)

FWIW, I'm asking myself why the Lisp community seems to agree that the
lowest levels of a CL persistent storage layer cannot be implemented
in CL itself.  Facts:

1. Plob is currently based on a C library and what you and others have
   proposed as replacements in this thread are also C/C++ libs.

2. AllegroStore, as offered by Franz, is based on ObjectStore.

3. Franz tries to replace AllegroStore with AllegroCache which is
   based on Berkeley DB.  (I've seen Jans Aasman demoing it in
   Amsterdam and he'll most likely repeat his talk at the European
   Common Lisp Meeting at the end of this month.  You can also read
   Bill Clementson's blog entries about it.)

4. Elephant is based on Berkeley DB.

Why is that?  I understand that this is something that's hard to do
right but is it so hard that even the largest CL vendor can't do it?
I don't expect a solution that's competitive with, say, Berkeley DB
from the very beginning, I'd be happy with something that's slower
than a pure C solution my a small constant factor.  I rarely have to
deal with billions of objects.

I also understand that a Lisp solution will probably involve
implementation-dependent and OS-specific code.  Well, so be it.  At
the moment you need the MOP and the FFI anyway, so the situation is
the same.

Would anyone care to comment?  If some of you with more Lisp
experience than I have think that Lisp is really not well-suited for
this task I'd be happy to learn about the reasons.

> For option 2., the POSTORE API is not that complicated. There are
> functions for allocating a persistent record with a quite simple
> structure (references to other persistent records and values [i.e.,
> no references]), access functions to these records by their `object
> id', and some administrative functions like flushing, creating a new
> persistent heap etc. You find the interface in the SH_* functions
> declared in plob-2.10/src/include/postore.h. All other
> databasespecific stuff (transactions, btrees etc.) has been
> implemented in Plob anyway in the layer above the persistent heap.

I'm confused here.  Let me summarize what I believe to understand and
please correct me where I'm wrong:

1. POSTORE offers efficient storage of (untyped?) records and is able
   to maintain and follow references between these records.  It also
   offers collection of unreferenced records.

2. POSTORE does not offer transactions or btrees both of which have
   been implemented in Plob instead.

3. Berkeley DB offers btrees and transactions but AFAIK no references
   between records and certainly no garbage collection.

How does that relate to your plan to replace POSTORE with Berkeley DB?

And, while I'm at it: Are transactions and btrees currently
implemented in C or in Lisp?

> Since my time for working on Plob is rather constrained, I would
> prefer option 2.

Well, I didn't necessarily expect /you/ to do all the work.  I was
contemplating to do something myself and for that I'm trying to sort
out what has to be done.

Thanks,
Edi.

Re: [PLOB] Upcoming release 2.11

[Heiko K. <hei...@gm...>]

Moin Edi,

Edi Weitz schrieb:

>On Wed, 06 Apr 2005 10:03:57 +0200, Heiko Kirschke <hei...@gm...> wrote:
>
>  
>
>>Second, to me it looks as if Plob should use its own underlying
>>persistent heap. There are several options to this:
>>
>>1. Abonding the mixed C-/LISP-approach of Plob, coding the
>>persistent heap in LISP. Would mean a major redesign and cost a lot
>>of effort.
>>    
>>
>
>But I seriously wonder if it's not worth it.  A pure CL solution will
>definitely be easier to debug and port, and it'll be easier to deliver
>applications based on it.  Plus, as Plob is an open source project, we
>should not forget that it's most likely much more fun for Lisp hackers
>to work on Lisp code than to work on C code... :)
>
>FWIW, I'm asking myself why the Lisp community seems to agree that the
>lowest levels of a CL persistent storage layer cannot be implemented
>in CL itself.  Facts:
>
Well, my reasoning is based more on, hmm, thinking about effort than
thinking that this would not be technically possible with a pure-CL
solution.

>1. Plob is currently based on a C library and what you and others have
>   proposed as replacements in this thread are also C/C++ libs.
>
>2. AllegroStore, as offered by Franz, is based on ObjectStore.
>
>3. Franz tries to replace AllegroStore with AllegroCache which is
>   based on Berkeley DB.  (I've seen Jans Aasman demoing it in
>   Amsterdam and he'll most likely repeat his talk at the European
>   Common Lisp Meeting at the end of this month.  You can also read
>   Bill Clementson's blog entries about it.)
>
>4. Elephant is based on Berkeley DB.
>
>Why is that?  I understand that this is something that's hard to do
>right but is it so hard that even the largest CL vendor can't do it?
>  
>
Since Franz is expert in Lisp Systems, I guess they wanted to buy some 
database instead
of coding it from scratch, as I assume also for the reason not having 
the efforts to code a
complete database system on their own. Expectations on a working 
database system
w.r.t to its ACID properties tend to be very high.

>I don't expect a solution that's competitive with, say, Berkeley DB
>from the very beginning, I'd be happy with something that's slower
>than a pure C solution my a small constant factor.  I rarely have to
>deal with billions of objects.
>
>I also understand that a Lisp solution will probably involve
>implementation-dependent and OS-specific code.  Well, so be it.  At
>the moment you need the MOP and the FFI anyway, so the situation is
>the same.
>  
>
Yes, indeed. And each new Lisp release introduces much more #+: es and 
#-: es into
Plob's code ...

>>For option 2., the POSTORE API is not that complicated. There are
>>functions for allocating a persistent record with a quite simple
>>structure (references to other persistent records and values [i.e.,
>>no references]), access functions to these records by their `object
>>id', and some administrative functions like flushing, creating a new
>>persistent heap etc. You find the interface in the SH_* functions
>>declared in plob-2.10/src/include/postore.h. All other
>>databasespecific stuff (transactions, btrees etc.) has been
>>implemented in Plob anyway in the layer above the persistent heap.
>>    
>>
>
>I'm confused here.  Let me summarize what I believe to understand and
>please correct me where I'm wrong:
>
>1. POSTORE offers efficient storage of (untyped?) records and is able
>   to maintain and follow references between these records.  It also
>   offers collection of unreferenced records.
>  
>
Yes to a large degree. On POSTORE level, the records are untyped but 
structured. There is
a structure imposed on each record, one half being references to other 
records, the other half
being uninterpreted values (see also chapter 7.4 of diplom.pdf provided 
within the Plob
distribution).

>2. POSTORE does not offer transactions or btrees both of which have
>   been implemented in Plob instead.
>  
>
Yes, and the POSTORE records are typed in the layer ,,above'' POSTORE.

>3. Berkeley DB offers btrees and transactions but AFAIK no references
>   between records and certainly no garbage collection.
>  
>
Yes. This could be added on top of  the Berkeley DB layer as an 
additional layer.

>How does that relate to your plan to replace POSTORE with Berkeley DB?
>  
>
It would mean to use Berkeley DB as a kind of persistent heap, too, i.e. 
impose a structure
on Berkeley DB's notion of database records. For Berkeley's additional 
functionality of
btrees and transactions this would mean that either the mechanisms of 
Plob are used
and Berkeley's would not be used at all, or that the Berkeley DB 
mechanisms are passed
through all layers up to the topmost API layer of Plob and remove these 
functionality
from the pure Plob layers.

>And, while I'm at it: Are transactions and btrees currently
>implemented in C or in Lisp?
>  
>
Completely C: in splobheap.c and splobbtree.c

>>Since my time for working on Plob is rather constrained, I would
>>prefer option 2.
>>    
>>
>Well, I didn't necessarily expect /you/ to do all the work.  I was
>contemplating to do something myself and for that I'm trying to sort
>out what has to be done.
>  
>
You're welcome! If you or someone else could show up with a solution of 
providing
some low level persistent layer in LISP, I would be happy to use it. 
Again, I'm thinking
in terms of effort, i.e. to create a good working solution in a moderate 
amount of time,
that is my reason d'etre. I see no technical obstacle for a pure LISP 
solution, and also
would prefer this kind of solution. IMHO, technical details on this can 
be discussed
offline.

(There is a minor technical reason I had in mind when using C for the 
lower layers: The
idea is that C is a kind of normed assembler, and that almost any other 
programming
language is able to connect somehow to C (i.e., by some kind of foreign 
function calls
resp. ,,native'' interface, e.g. Java's JNI). So, when using C for 
Plob's lower layers,
the upper layers could have been done in some other programming language 
besides
LISP. Since this has never been carried out, it makes sense to forget 
about this idea.)

Viele Grüße,
Heiko

Re: [PLOB] Upcoming release 2.11

[Edi W. <ed...@ag...>]

On Thu, 07 Apr 2005 15:10:50 +0200, Heiko Kirschke <hei...@gm...> =
wrote:

> Well, my reasoning is based more on, hmm, thinking about effort than
> thinking that this would not be technically possible with a pure-CL
> solution.

OK, that sounds better to my ears.  I'm still wondering why seemingly
nobody has done a pure Lisp solution since the days of Statice,
though.

> Since Franz is expert in Lisp Systems, I guess they wanted to buy
> some database instead of coding it from scratch, as I assume also
> for the reason not having the efforts to code a complete database
> system on their own. Expectations on a working database system w.r.t
> to its ACID properties tend to be very high.

But one shouldn't underestimate the marketing value.  Franz is in the
business of selling Lisp and basing their new flagship database
product on a C library is a bit like "Lisp is quite good for certain
things but for the real stuff you have to use C of course."

Oh, and thanks for all the technical explanations that I have snipped.

> You're welcome!

Fine.  Don't hold your breath, though.  This is something I've been
thinking about for a long time but I can only do it in my limited
spare time.  (Although finally I'll hopefully benefit from it in my
professional work as well.)

> If you or someone else could show up with a solution of providing
> some low level persistent layer in LISP, I would be happy to use
> it. Again, I'm thinking in terms of effort, i.e. to create a good
> working solution in a moderate amount of time, that is my reason
> d'etre. I see no technical obstacle for a pure LISP solution, and
> also would prefer this kind of solution. IMHO, technical details on
> this can be discussed offline.

OK, we can of course meet for a coffee... :)

But don't you think this mailing list is the right forum for this kind
of discussion?

Beste Gr=FC=DFe,
Edi.

Re: [PLOB] Upcoming release 2.11

[Heiko K. <hei...@gm...>]

Moin Edi,

Edi Weitz schrieb:

>>Since Franz is expert in Lisp Systems, I guess they wanted to buy
>>some database instead of coding it from scratch, as I assume also
>>for the reason not having the efforts to code a complete database
>>system on their own. Expectations on a working database system w.r.t
>>to its ACID properties tend to be very high.
>>    
>>
>
>But one shouldn't underestimate the marketing value.  Franz is in the
>business of selling Lisp and basing their new flagship database
>product on a C library is a bit like "Lisp is quite good for certain
>things but for the real stuff you have to use C of course."
>
>Oh, and thanks for all the technical explanations that I have snipped.
>  
>
Well, I think that if an API does not constrain the possibilities of the 
language
using the API, it fairly does not mind how this API is implemented (is 
this an
`extensional interface'?). I've tried to use this approach when coding Plob.

>But don't you think this mailing list is the right forum for this kind
>of discussion?
>  
>
*This* kind of discussion is ok. What I meant is I don't want to discuss 
here e.g.
technical details about the persistent heap layers API or things like 
that. `Strategic'
discussions like the ones we had about how the further development of 
Plob could be
done are necessary and therefore completely ok (as long as the readers 
don't start to
unsubscribe ...)

Viele Grüße,
Heiko

[PLOB] Re: Upcoming release 2.11

[Klaus H. <kl...@ha...>]

Heiko Kirschke <hei...@gm...> writes:

> Moin Edi,
>
> Edi Weitz schrieb:
>
>>>I have still plans to replace PLOB's underlying library POSTORE with
>>>Berkely DB, but unfortunately did not have the time to do so.
>>>    
>>>
>>It'd be great if you could replace the underlying store with something
>>else.  I wonder, however, if Berkeley DB is the right solution.
>>Again, because of licensing (or rather pricing) issues.  See my email
>>to the Elephant mailing list:
>>
>>  <http://common-lisp.net/pipermail/elephant-devel/2005-March/000121.html>
>>  
>>
> Ok, reading this means that the Berkeley DB license is not suitable
> for Plob. Sigh.
>
>>Could you describe in a few words which features have to be supported
>>by an alternative back-end?
>>
> The requirements are not that high.
>
> First, I know of a successor of POSTORE called `Lumberjack' with the
> same API. Unfortunately, it looks as if the work on Lumberjack has
> stuck a bit, since the author(s) told me that it is not yet ready for
> putting it into the public domain. I will ask them again.
>
> Second, to me it looks as if  Plob should use its own underlying
> persistent heap. There are several options to this:
>
> 1. Abonding the mixed C-/LISP-approach of Plob, coding the persistent
> heap in LISP. Would mean a major redesign and cost a lot of effort.
>
> 2. Replace the POSTORE lib by an own implementation of a persistent
> heap, again in C. Would mean only a redesign on Plob's lower levels,
> with maybe moderate efforts. I cannot estimate what it would mean to
> replace the POSTORE lib by something coded in LISP, but I'm afraid
> that this would mean that I've to drop the idea of using Plob randomly
> in server or serverless mode and to decide on supporting only exactly
> one of these modes.
>
> For option 2., the POSTORE API is not that complicated. There are
> functions for allocating a persistent record with a quite simple
> structure (references to other persistent  records and values [i.e.,
> no references]), access functions to these records by their `object
> id', and some administrative functions like flushing, creating a new
> persistent heap etc. You find the interface in the SH_* functions
> declared in plob-2.10/src/include/postore.h. All other
> databasespecific stuff (transactions, btrees etc.) has been
> implemented in Plob anyway in the layer above the persistent heap.
>
> Since my time for working on Plob is rather constrained, I would
> prefer option 2.
>
> Viele Grüße from Hamburg, too,
> Heiko
>
> Heiko Kirschke
> Achtern Barg 48
> D-22885 Barsbüttel
> Tel.: +49 40 225497
>
>
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now.
> http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Hi Heiko --

First of all, I think this is really good news.  I tried out PLOB!
about a year ago - worked really well.  However, due to the licensing
issues and its -- apparent, it now turns out! -- status as semi-dead,
I abandoned exploration beyond the initial try-out.  It would be great
to see PLOB! revived.

I agree with Edi wrt. the licensing issues from Sleepycat - I think it
is a showstopper; it is for me/us, at least.  Even if you can use it
for private and open source projects, not being able to use it
professionally means that it becomes really hard to justify dealing
with the learning curve.

I haven't looked much at PLOB! source, but perhaps an SQLite backend
is a possibility? 

regards,

-Klaus.

Re: [PLOB] Re: Upcoming release 2.11

[Edi W. <ed...@ag...>]

On Wed, 06 Apr 2005 10:31:09 +0200, Klaus Harbo <kl...@ha...> wrote:

> I haven't looked much at PLOB! source, but perhaps an SQLite backend
> is a possibility?

I've played a bit with SQLite recently and while it's easily deployed
(only one DLL and one database file) it seems to have serious
performance problems under certain circumstances.  For example, if you
have hundreds of database updates and wrap them all into one
transaction then it's quite fast.  If, however, each update has its
own transaction it crawls.  I'd investigate these things before using
it in earnest.

Cheers,
Edi.

Re: [PLOB] Upcoming release 2.11

[Peter D. <pet...@ni...>]

Hi,

You might have a look at Texas, a persistent storage facility that has been 
integrated with RScheme. I recall speaking with the author, Paul Wilson a few 
years ago about its use in common lisp. He thought it would not be hard to 
integrate.

http://www.cs.utexas.edu/users/oops/papers.html#texas



On Wednesday 06 April 2005 03:33, Edi Weitz wrote:
> Hi!
>
> On Wed, 06 Apr 2005 08:52:13 +0200, Heiko Kirschke <hei...@gm...> 
wrote:
> > I'm preparing release 2.11 of Plob. The new release will include a
> > more stable version of Plob's server code (i.e., persistent heap)
> > for Windows; furthermore, I'm updating Plob so that it will run on
> > the actual LISP versions of LispWorks (4.4.x) and Franz' ACL (7.x).
>
> Cool, thanks.
>
> > I have still plans to replace PLOB's underlying library POSTORE with
> > Berkely DB, but unfortunately did not have the time to do so.
>
> It'd be great if you could replace the underlying store with something
> else.  I wonder, however, if Berkeley DB is the right solution.
> Again, because of licensing (or rather pricing) issues.  See my email
> to the Elephant mailing list:
>
>   <http://common-lisp.net/pipermail/elephant-devel/2005-March/000121.html>
>
> Could you describe in a few words which features have to be supported
> by an alternative back-end?
>
> Thanks for all on your work on this.
>
> Greetings from Hamburg,
> Edi.
>
>
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now.
> http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
> _______________________________________________
> Plob-discussion mailing list
> Plo...@li...
> https://lists.sourceforge.net/lists/listinfo/plob-discussion

-- 
Best Regards,
 - Peter

Re: [PLOB] Upcoming release 2.11

[Heiko K. <hei...@gm...>]

All,

Ala...@me... schrieb:

>I wonder if the ISAM layer code of Postgresql, or something
>like SQLITE, could be made to serve.
>
Actually, I'm also thinking of something like ISAM. Using some other 
in-memory DB (like FastDB or Texas)
might bring relief to the license discussion, but won't be a real 
technical advantage. I'll take a look
at the ISAM stuff of Postgres, esp. since Plob uses already the 
hierarchical locking algorithm
of Postgres :-) Some lower layer of SQLite might also be promising.

>  I've also seen something
>called dxstore, which seemed good, but appears to have been
>abandoned by its author.
>  
>
Looks so, the last version is of 2000-06-12

>In any case, I think freeing PLOB! of it's licensing shackles
>is a good idea which will greatly increase its use.
>  
>
Well, I'll do my very best.

Viele Grüße,
Heiko

[PLOB] Re: Upcoming release 2.11

[Klaus H. <kl...@ha...>]

Heiko Kirschke <hei...@gm...> writes:

> Well, I think that if an API does not constrain the possibilities of
> the language using the API, it fairly does not mind how this API is
> implemented (is this an `extensional interface'?). I've tried to use
> this approach when coding Plob.
>
>>But don't you think this mailing list is the right forum for this
>> kind
>>of discussion?
>>  
>>
> *This* kind of discussion is ok. What I meant is I don't want to discuss
> here e.g.  technical details about the persistent heap layers API or
> things like that. `Strategic' discussions like the ones we had about
> how the further development of Plob could be done are necessary and
> therefore completely ok (as long as the readers don't start to
> unsubscribe ...)
>
> Viele Grüße, Heiko
>

FWIW, I'm quite interested in this subject as well, so I would like to
see the discussion stay 'on the list', at least until the discussions
get to a very high volume!

-Klaus.