[ZooLib-dev] Re: [ZooLib-checkins] CVS: zoolib/src_other/tuplebase ZTS_BlockStore.cpp,1.4,NONE ZTS_BlockStore.h,1.3,NONE

[Andrew G. <ag...@em...>]

Hi. Nice to hear from you again. How did you get on in the end with 
xcode and zoolib?

On Aug 18, 2004, at 23:22, Richard Clark wrote:

> The only thing to look out for is that BerkeleyDB breaks compatibility 
> with old data stores whenever they upgrade the software. There needs 
> to be as good dump/restore capability, perhaps as a serialization 
> mechanism in ZBlockStore.

Where things stand at the moment with the tuplebase stuff is that the 
application-level API is provided by ZTB, ZTxn, ZTBSpec, ZTBQuery, 
ZTBIter and ZTuple. That stuff has been stable even in most of the 
details for the last 8 months or so, and the fundamental stucture has 
been unchanged for the last 20 months.

The implementation resides in subclasses of ZTBRep. There are two of 
those right now, ZTBRep_Client and ZTBRep_TS. ZTBRep_Client uses a 
ZStreamR/ZStreamW pair to talk to an instance of ZTBServer which itself 
talks to an instance of a ZTBRep subclass. Because the ZTBRep API is 
async the ZTBRep_Client/ZTBServer combo is able to be more efficient 
than if it were simply proxying calls across a comms link.

ZTBRep_TS is where most of the real tuplebase work happens, but it 
delegates the actual storage and basic searching of tuples to an 
instance of a ZTS subclass. The only extant subclass of ZTS is ZTS_RAM, 
which keeps all the tuples in a map<> and maintains a number of indices 
over the tuples. ZTS_RAM can dump the tuples it holds to a stream in 
binary format, or to a strim in parseable text format. It can also load 
its tuples from a stream or a strim, regenerating its indices as it 
does so.

The ZTS API is where we could make use of BerkeleyDB, mainly so the 
indices are persistent. Loading the tuples themselves is pretty quick; 
on my laptop it does at about 10,000/second in binary format, 
1,000/second in text format. The general index code runs about 1,000 
tuples per index per second. So If you have four indices generating the 
index takes 80% of the load up time for tuples coming from a strim, 99% 
for a stream.

> One web host I know basically tells people not to use BerkeleyDB for 
> their blogs and to configure for MySQL instead.

If the BerkeleyDB format is that volatile we can continue to store 
tuples in a flat stable file format (maybe some log-structured thing) 
and only use BerkeleyDB for the indices.

A+
-- 
Andrew Green                           mailto:ag...@em...
Electric Magic Co.               vox: +1 (831) 621 1822