Re: [Ocaml-lib-devel] Future of ExtLib

[Brian H. <bh...@sp...>]

On Fri, 9 Apr 2004, Nicolas Cannasse wrote:

> > 1) sinks and sources.  
> > 2) filters.  
> Theses two are now reality in the IO module.

This is good- these two were the "core" of the idea, the rest was 
decoration.

> For example, I managed to wrap CamlZip module on an IO and - without
> changing a line of my code - I can now read/write over normal or compressed
> files. Here's the code that's converting a standard IO into a
> compressed/uncompressed one. I think it's worth including into the ExtLib
> but the problem is it rely on another module (CamlZip, by Xavier Leroy)
> which itself contains C stubs using the ZLib. I think we need an additional
> folder into the ExtLib called "tools" that will be useful parts of code -
> yet not installed because of external dependencies.
> What do you think about the idea ?

Actually, this is a deep problem, and not limited to just compression.  I
could see reimplementing zlib in pure Ocaml.  But the #2 thing to do is
encryption/decryption.  For which you should not only call out to C, but
have the C call hand tuned assembly.
 
20 cycles/byte encryption speed (approx. the speed of a good symmetric key
crypto system like AES or Twofish) sounds real good, until you realize
that this means 10 milliseconds to encrypt a megabyte of data on a 2GHz
machine.  Similiarly, I'm willing to be that RSA encryption using GMP will
be signifigantly faster than using the Ocaml bignum library.  Every cycle
counts in this special case- even C is too slow.

Compression we might want to reimplement in Ocaml.  Reimplementing the 
crypto libraries in pure Ocaml, while theoretically possible, would impose 
severe performance limits.

So, there are three possibilities:

	- Stick with implementing everything in Ocaml.  This gives us 
portability and library independence (you don't need zlib nor gmp 
installed) at the cost of performance

	- Allow C/ASM callouts for performance

	- Provide both, and some mechanism for choosing between them (with 
the attendent problems)

-- 
"Usenet is like a herd of performing elephants with diarrhea -- massive,
difficult to redirect, awe-inspiring, entertaining, and a source of
mind-boggling amounts of excrement when you least expect it."
                                - Gene Spafford 
Brian