Re: [Jdbm-general] compression & encryption

[Kris L. <kl...@im...>]

Alex,

BTree index key compression is very effective if there is numerous
common prefix(s).  Personal experience with my applications have
not showen that key compression has been worth the work to put
that feature in.

On the other hand, DB compression would appear to be useful. 
In my case, the objects are being serialized into XML, compression
of the byte[] might reduce the DB.  I use the word "appear" since
I have not done any tests to confirm my theory.

As for encryption, simple standard encryption is best for performance.
Also public key cyrpto is just over kill for my needs.

Thanks for letting me put in my 2 cents.

Regards,
Kris




Alex Boisvert wrote:

>
> Kris,
>
> Very cool that you're using JDBM 24/7!   So am I!
>
> Just to clarify that BTree index key compression is not your typical 
> Hoffman-style compression.
>
> Key compression refers to the inherent redundancy in consecutive keys 
> in an index.   Because keys are ordered, they generally share a common 
> prefix.  And the bigger the index, the longer the prefix.
>
> So if you have the following keys on a BPage:
>
> AAAABCDEF, AAAACDEFG, AAAADEFGH
>
> then you would "compress" the keys by using a prefix (usually stored 
> on the BPage) like this:
>
> PREFIX = AAAA
> KEYS = *BCDEF, *CDEFG, *DEFGH
> (where * represents the common prefix)
>
> This typically works much better than Hoffman-style compression 
> applied to a single key and is relatively cheap CPU-wise.
>
> You can also store the prefix on a parent BPage and achieve even 
> greater compression ratios.
>
> As to encryption of DB data, I guess we could provide some kind of 
> interceptor at the RecordFile level.   Did you have something specific 
> in mind?   Standard encryption or public key crypto?
>
> alex
>
>
> Kris Leite wrote:
>
>> If you would allow my vote.
>>
>> 1) BTree index key compression  =  -1 - see itch #1 below.
>> 2) Implement collections interfaces to follow Java idioms = 0
>> 3) Improve concurrency and transaction management         = +1
>> 4) Improve I/O -- think java.nio  =  -1 - Currently preforms very 
>> nicely.
>>
>> I have a couple of itchs.  <grin>
>>
>> 1)  Compress before storing  (could reduce need for key compression)
>> 2)  Encryption of DB data (after compression)  (Why? to keep end 
>> users of
>> our applications from accessing data that they are not allowed to see.)
>>
>> Thanks,
>> Kris
>>
>> PS - I have implemented JDBM very successfully in two different 
>> applications that
>> run 24/7/365.   Thank you very much for a great piece of code.
>
>
> .
>