Re: [Firebird-devel] RFC: Adding a cryptographic library for use in firebird

[Mark O'D. <mar...@gm...>]

Hi Adriano

Thanks for replying, below turned into an essay, but hopefully it
answers most of your questions.  I will also, in a few days, build and
put up an experimental test copy, for people to download and have a
look at.

On Sat, Nov 22, 2008 at 11:06 PM, Adriano dos Santos Fernandes
<adr...@uo...> wrote:
> Mark O'Donohue wrote:
>> Hi All
>>
> First, AFAIK strong encription is not allowed on some countries. How do
> we deal with it?
>

Oh, you mean France :-).

http://www.praxagora.com/andyo/ar/crypto_model.html

I am not sure, in the old days  you used to have to offer a build with
encryption and without, but that doesn't seem to happen much any more,
the crypto bits firefox are as strong as openssl, but they don't
offers a 40 bit or no encryption download any more.

But presumably to comply with this law, people in France are already
building their linux boxes, and their apache webservers,  from
scratch, and do not use firefox, :-).   And presumably they also don't
use mysql which also includes openssl.
http://dev.mysql.com/doc/refman/5.0/en/encryption-functions.html   But
I would be interested in Paul B, and Paul R's view on this,since they
live in France, technically I guess they would not be able to connect
to sourceforge using ssh.

The other serious issue on this front is exporting from the USA, again
that has relaxed over the years, I remember when all of Debian was
downloadable from the US, except one file that you had to get from
Finland,  and technically it was ok to fax, Bruce Schneier's
appendix's in Applied Cryptography ,
http://www.schneier.com/book-applied.html, but illegal to download
them.   These days, we download our redhat binaries (+openssl) without
batting an eyelid.  This is allowable it seems because since openssl
can be obtained outside the states already, then it isn't breaking any
laws.

But if you build your own crypto library in the US, you can run into
trouble, as Phil Zimmermann found out, with PGP,
http://www.virtualschool.edu/mon/Crypto/PGPPrivacyWars.html.  The
acceptable outcome for everyone is rather childish and pointless, the
court doesn't know how PGP escaped the US boundaries, yes it was
placed on an ftp site, but they are happy that Phil wasn't personally
involved in exporting it, and now they offer a crippled version for
download from MIT.

But, I think we should do what everyone else does, use a library that
is available world wide (and openssl is the one I suggest) rather than
trying to tinker or copy individual bits of code into our source base,
the export boundaries and problems for US code are less, but they are
not gone.


> What is the library source and binary size? Would they will be required
> or FB can work without them? How it work without them?
>

The latest source download is about 4.5meg in size.
http://www.openssl.org/source/.

For Linux, Unix, I would suggest linking with the OS provided openssl
shared libraries, as they are part of the standard OS distribution
anyway, for windows it may be preferable to link the libraries
statically.

The library sizes on windows are:
                 libeay32.dll 996K
                 ssleay32.dll 196k

          or statically:
                 libeay32.lib  3,839k
                 ssleay32.lib 600k

not, sure why static ones are larger, maybe I did a debug build.  I
will do an experimental Firebird build, and see how it changes the
binary size.

But, just to add,  we do get a  lot of value in these openssl
packages, not just the core crypto routines, but all the routines for
certificate handling, pkcs7 signatures, pem encoding, even proper
asn.1 parsing  (asn.1 does the same mistakes as xml but they just did
it earlier).


> I don't think it should be done for v2.5, as all these benefits will not
> be achieved, and there are important points (questions above, plugin
> discussions) to resolve.
>

Actually, the basics I am suggesting should work fairly quickly, and
be fairly stable.  Interestingly, the code for database encryption is
also already there, thanks to Dmitry back in 2003, I can enable his
code via my routines fairly easily, the only remaining issue with
Dmity's code was how to pass in the encryption key in, but that is
also achievable within a reasonable timeframe.

Whatever tree the crypto library is added to it should always be a
conditional compilation.  the FBCrypto layer I am suggesting is a good
stopping point, it will perform the function or throw an exception if
the implementation doesn't exist.

Should it be included in 2.5 tree? I am not sure, since it would be
conditional it could be done without much impact,  but I suspect it
all depends how long it is going to take for 2.5 to get to a final
release, the release dates when you are relying on people to put in
their own time for free,  do tend to slip a bit.

However, I must admit, which tree, was in my mind  when I looked at
SysFunctions.cpp, and really bit my tongue and bypassed changing this:

      const SysFunction* SysFunction::lookup(const Firebird::MetaName& name)
     {
	    for (const SysFunction* f = functions; f->name.length() > 0; ++f)
 	   {
		if (f->name == name)
			return f;
	   }

	return NULL;
    }

to a HashMap<Key,Value> style lookup (I see Alex has added some
GenericMap<Key, Value> templates)   It would only take about 2hrs of
refactoring, and the chance of a mistake is minimal, but isn't the
sort of change you want to do prior to a release.   If it goes into
3.0, then I will change this in SysFunctions.cpp, and also rework the
makeXXX, evlXXX, and setParmsXXX functions as methods of classes.


>>
> For hashing functions, I think the HASH system function should be
> extended with a new parameters. But to describe the return type
> correctly, the algorithm should be evaluated at compilation phase.
>

I can see how that makes some sense, I haven't started on those.   The
current Hash is a different beast, it is like a CRC check, and not
very strong from a cryptographic point of view.   But I can see how
changing to Hash(digestName, data)  and using hash('CRC', data) or
hash('SHA256', data)  could work.  Overall I think its probably best
to leave Hash alone and create separate functions, but I don't have a
strong opinion on it.

> Why ByteArray? We already have array classes.

Yes, good question, I really did want to use a standard existing class
for that, and tried fairly hard to.

I built my first version of ByteArray as an extension of Array<> but
then it didn't offer good support for they type of functions I wanted,
particularly reference counting, and the methods I wanted all seemed
to be part of fb_string.  So I looked at fb_string.h and
AbstractString was based on char*, and  so I tried converting that to
AbstractString<charT> where it would have a common base, but it fell
apart at the next level at StringComparator and StringType stage,
there was also a lot of special stuff going with ensuring the \0 on
the end and calls to strlen() which if mixed with unsigned chars,
someone would make an easy mistake and cause trouble.  I came to the
conclusion that overall, a combined base, would be very messy and
troublesome to maintain.  It also wasn't going to do reference
counting.

Finally I gave up, and implemented what I wanted, although based on
the functions in fb_string, and internally the private implementation
class ByteArray_Impl, I suspect it will, once I get some advice on
integrating Memory pools properly it probably will use Array<unsigned
char>.    ( I must admit I dislike Memory pools, because they cripple
and make more complex what would otherwise be simple C++ code - I
haven't followed the debate too closely, but those Memory Pools had
better be giving good return for all the trouble they cause!).

> Also, won't the crypto functions works in chunks of bytes?
> Does it works in chunks for blobs?
>

The routines I have added works on the whole blocks of data, so it
reads a whole blob, encrypts it, then writes a new blob, that is a
good starting point.  Internally the underlying implementation uses
ssleay BIO*  streams, and a source or destination BIO can be either a
memory buffer (growable) or a socket.  Certainly this is needed for
SSL communication, and I am sure with a bit of tinkering we can create
a BIO filter that works with blobs - but it is not there yet.

>>
> I do not like the need to cast. This is another point that could be
> resolved in v3.
>

I agree, I don't like casting either, we need a vanilla decrypt as a
starting point.  But in the system function level, it is possible to
do a bit more, Alex suggested adding a type-byte at the front, so the
decrypt knew what type the object started life as, that is worth
exploring.

Hope that is enough, I will work on getting an experimental build
available for download - so people can start finding bugs or problems
:-).

Cheers - Mark