Re: [PLOB] Upcoming release 2.11

[Edi W. <ed...@ag...>]

Moin!

On Wed, 06 Apr 2005 10:03:57 +0200, Heiko Kirschke <hei...@gm...> wrote:

> Second, to me it looks as if Plob should use its own underlying
> persistent heap. There are several options to this:
>
> 1. Abonding the mixed C-/LISP-approach of Plob, coding the
> persistent heap in LISP. Would mean a major redesign and cost a lot
> of effort.

But I seriously wonder if it's not worth it.  A pure CL solution will
definitely be easier to debug and port, and it'll be easier to deliver
applications based on it.  Plus, as Plob is an open source project, we
should not forget that it's most likely much more fun for Lisp hackers
to work on Lisp code than to work on C code... :)

FWIW, I'm asking myself why the Lisp community seems to agree that the
lowest levels of a CL persistent storage layer cannot be implemented
in CL itself.  Facts:

1. Plob is currently based on a C library and what you and others have
   proposed as replacements in this thread are also C/C++ libs.

2. AllegroStore, as offered by Franz, is based on ObjectStore.

3. Franz tries to replace AllegroStore with AllegroCache which is
   based on Berkeley DB.  (I've seen Jans Aasman demoing it in
   Amsterdam and he'll most likely repeat his talk at the European
   Common Lisp Meeting at the end of this month.  You can also read
   Bill Clementson's blog entries about it.)

4. Elephant is based on Berkeley DB.

Why is that?  I understand that this is something that's hard to do
right but is it so hard that even the largest CL vendor can't do it?
I don't expect a solution that's competitive with, say, Berkeley DB
from the very beginning, I'd be happy with something that's slower
than a pure C solution my a small constant factor.  I rarely have to
deal with billions of objects.

I also understand that a Lisp solution will probably involve
implementation-dependent and OS-specific code.  Well, so be it.  At
the moment you need the MOP and the FFI anyway, so the situation is
the same.

Would anyone care to comment?  If some of you with more Lisp
experience than I have think that Lisp is really not well-suited for
this task I'd be happy to learn about the reasons.

> For option 2., the POSTORE API is not that complicated. There are
> functions for allocating a persistent record with a quite simple
> structure (references to other persistent records and values [i.e.,
> no references]), access functions to these records by their `object
> id', and some administrative functions like flushing, creating a new
> persistent heap etc. You find the interface in the SH_* functions
> declared in plob-2.10/src/include/postore.h. All other
> databasespecific stuff (transactions, btrees etc.) has been
> implemented in Plob anyway in the layer above the persistent heap.

I'm confused here.  Let me summarize what I believe to understand and
please correct me where I'm wrong:

1. POSTORE offers efficient storage of (untyped?) records and is able
   to maintain and follow references between these records.  It also
   offers collection of unreferenced records.

2. POSTORE does not offer transactions or btrees both of which have
   been implemented in Plob instead.

3. Berkeley DB offers btrees and transactions but AFAIK no references
   between records and certainly no garbage collection.

How does that relate to your plan to replace POSTORE with Berkeley DB?

And, while I'm at it: Are transactions and btrees currently
implemented in C or in Lisp?

> Since my time for working on Plob is rather constrained, I would
> prefer option 2.

Well, I didn't necessarily expect /you/ to do all the work.  I was
contemplating to do something myself and for that I'm trying to sort
out what has to be done.

Thanks,
Edi.