Re: [PLOB] Upcoming release 2.11

[Heiko K. <hei...@gm...>]

Moin Edi,

Edi Weitz schrieb:

>On Wed, 06 Apr 2005 10:03:57 +0200, Heiko Kirschke <hei...@gm...> wrote:
>
>  
>
>>Second, to me it looks as if Plob should use its own underlying
>>persistent heap. There are several options to this:
>>
>>1. Abonding the mixed C-/LISP-approach of Plob, coding the
>>persistent heap in LISP. Would mean a major redesign and cost a lot
>>of effort.
>>    
>>
>
>But I seriously wonder if it's not worth it.  A pure CL solution will
>definitely be easier to debug and port, and it'll be easier to deliver
>applications based on it.  Plus, as Plob is an open source project, we
>should not forget that it's most likely much more fun for Lisp hackers
>to work on Lisp code than to work on C code... :)
>
>FWIW, I'm asking myself why the Lisp community seems to agree that the
>lowest levels of a CL persistent storage layer cannot be implemented
>in CL itself.  Facts:
>
Well, my reasoning is based more on, hmm, thinking about effort than
thinking that this would not be technically possible with a pure-CL
solution.

>1. Plob is currently based on a C library and what you and others have
>   proposed as replacements in this thread are also C/C++ libs.
>
>2. AllegroStore, as offered by Franz, is based on ObjectStore.
>
>3. Franz tries to replace AllegroStore with AllegroCache which is
>   based on Berkeley DB.  (I've seen Jans Aasman demoing it in
>   Amsterdam and he'll most likely repeat his talk at the European
>   Common Lisp Meeting at the end of this month.  You can also read
>   Bill Clementson's blog entries about it.)
>
>4. Elephant is based on Berkeley DB.
>
>Why is that?  I understand that this is something that's hard to do
>right but is it so hard that even the largest CL vendor can't do it?
>  
>
Since Franz is expert in Lisp Systems, I guess they wanted to buy some 
database instead
of coding it from scratch, as I assume also for the reason not having 
the efforts to code a
complete database system on their own. Expectations on a working 
database system
w.r.t to its ACID properties tend to be very high.

>I don't expect a solution that's competitive with, say, Berkeley DB
>from the very beginning, I'd be happy with something that's slower
>than a pure C solution my a small constant factor.  I rarely have to
>deal with billions of objects.
>
>I also understand that a Lisp solution will probably involve
>implementation-dependent and OS-specific code.  Well, so be it.  At
>the moment you need the MOP and the FFI anyway, so the situation is
>the same.
>  
>
Yes, indeed. And each new Lisp release introduces much more #+: es and 
#-: es into
Plob's code ...

>>For option 2., the POSTORE API is not that complicated. There are
>>functions for allocating a persistent record with a quite simple
>>structure (references to other persistent records and values [i.e.,
>>no references]), access functions to these records by their `object
>>id', and some administrative functions like flushing, creating a new
>>persistent heap etc. You find the interface in the SH_* functions
>>declared in plob-2.10/src/include/postore.h. All other
>>databasespecific stuff (transactions, btrees etc.) has been
>>implemented in Plob anyway in the layer above the persistent heap.
>>    
>>
>
>I'm confused here.  Let me summarize what I believe to understand and
>please correct me where I'm wrong:
>
>1. POSTORE offers efficient storage of (untyped?) records and is able
>   to maintain and follow references between these records.  It also
>   offers collection of unreferenced records.
>  
>
Yes to a large degree. On POSTORE level, the records are untyped but 
structured. There is
a structure imposed on each record, one half being references to other 
records, the other half
being uninterpreted values (see also chapter 7.4 of diplom.pdf provided 
within the Plob
distribution).

>2. POSTORE does not offer transactions or btrees both of which have
>   been implemented in Plob instead.
>  
>
Yes, and the POSTORE records are typed in the layer ,,above'' POSTORE.

>3. Berkeley DB offers btrees and transactions but AFAIK no references
>   between records and certainly no garbage collection.
>  
>
Yes. This could be added on top of  the Berkeley DB layer as an 
additional layer.

>How does that relate to your plan to replace POSTORE with Berkeley DB?
>  
>
It would mean to use Berkeley DB as a kind of persistent heap, too, i.e. 
impose a structure
on Berkeley DB's notion of database records. For Berkeley's additional 
functionality of
btrees and transactions this would mean that either the mechanisms of 
Plob are used
and Berkeley's would not be used at all, or that the Berkeley DB 
mechanisms are passed
through all layers up to the topmost API layer of Plob and remove these 
functionality
from the pure Plob layers.

>And, while I'm at it: Are transactions and btrees currently
>implemented in C or in Lisp?
>  
>
Completely C: in splobheap.c and splobbtree.c

>>Since my time for working on Plob is rather constrained, I would
>>prefer option 2.
>>    
>>
>Well, I didn't necessarily expect /you/ to do all the work.  I was
>contemplating to do something myself and for that I'm trying to sort
>out what has to be done.
>  
>
You're welcome! If you or someone else could show up with a solution of 
providing
some low level persistent layer in LISP, I would be happy to use it. 
Again, I'm thinking
in terms of effort, i.e. to create a good working solution in a moderate 
amount of time,
that is my reason d'etre. I see no technical obstacle for a pure LISP 
solution, and also
would prefer this kind of solution. IMHO, technical details on this can 
be discussed
offline.

(There is a minor technical reason I had in mind when using C for the 
lower layers: The
idea is that C is a kind of normed assembler, and that almost any other 
programming
language is able to connect somehow to C (i.e., by some kind of foreign 
function calls
resp. ,,native'' interface, e.g. Java's JNI). So, when using C for 
Plob's lower layers,
the upper layers could have been done in some other programming language 
besides
LISP. Since this has never been carried out, it makes sense to forget 
about this idea.)

Viele Grüße,
Heiko