Menu
▾
▴
firebird-devel
|
From: Mark O'D. <mar...@gm...> - 2008-11-22 00:30:18
|
Hi All
I would like to add a general cryptographic library of functions for
usage in Firebird.
There are many Firebird enhancements that have been suggested:
encrypted disk images; secure client connection handshake; encrypted
backups (aka Borland Interbase); even digitally encrypted and signed
BLR routines; and native SSL engine support. A key enabler of those
features is the availability of a general set of cryptographic
functions in the engine. I would like to add an FBCrypto interface
class that provides those functions.
The FBCrypto class adds a C++ friendly layer over the top of the
OpenSSL libraries, and provides access to the cryptographic routines
without exposing the openssl implementation details. As a first step
in introducing the library, I also propose adding a number of SQL
system functions, that make use of this library to perform
encryption/hashing and general base64 encoding and decoding.
The interface classes are described here:
http://www.eftel.com.au/~mark.odonohue/FBCrypto.html
http://www.eftel.com.au/~mark.odonohue/ByteArray.html
Here is a sample usage, taken from the prototype encrypt system
function implementation:
Encryptor enc = FBCrypto::getEncryptor();
enc.setCipher(cipherName);
enc.setPassphrase(passphrase);
enc.initEncrypt();
ByteArray encResult = enc.encrypt(rawData);
And the suggested additional SQL system functions are:
encrypt(cipherName, keyphrase, data)
decrypt(cipherName, keyphrase, data)
createdigest(digestName, data)
verifydigest(digestName, digest, data)
base64encode(data)
base64decode(b64string)
hexencode(data)
hexdecode(hexstring)
With example usage:
update table1 set encblob = encrypt('AES128','secret', datablob))
update table1 set signature = createdigest('SHA256', datablob))
select base64encode(encblob) from table1;
Currently only the first two functions are implemented, although all
the underlying C++ functions exist. The encrypt function takes string
or blob data, and returns binary string or binary blob, depending on
the input data type. The return result for decrypt, is a binary
string or binary blob, and will need to be cast to the appropriate
type, although this could use octet strings.
To be sure I it wasn't going to be promise-ware, I have built, on both
linux and win32, a basic working model based on fb2.5 head. It is
conditionally compiled, so can fit into the codebase without much
disruption while it is being worked on. Most of the tricky crypto
code is fine, since its based on openssl, there is some finalization
to do with how the pimpl delegated implementation uses memory pools,
and more testing required. But this is at a good point to have a
discussion, and see where it best fits in, and the feedback and
discussion will likely influence where it will go from here.
Comments most welcome.
Cheers - Mark
BTW: I also have a similar set of C++ routines, that do x509/RSA
certificate handling using the same approach, but their interface not
finalized as yet.
|
|
From: Mark O'D. <mar...@gm...> - 2008-11-22 01:02:40
|
Alex wrote previously: > - any plugin (shared library) should be with > minimum efforts linkable with firebird statically. The primary reason for > such a requirement is our undocumented, but used and loved by people, single > file copy client install on windows. I.e. each client-side plugin should be > easily linked statically inside fbclient.dll. The openssl libraries can be built either dynamicly or statically. On windows interestingly enough, openssl is often linked in statically, to avoid conflict with other products that also use openssl, but may have loaded different versions. I currently have both built for win32. Cheers - Mark |
|
From: Adriano d. S. F. <adr...@uo...> - 2008-11-22 12:07:58
|
Mark O'Donohue wrote: > Hi All > > I would like to add a general cryptographic library of functions for > usage in Firebird. > First, AFAIK strong encription is not allowed on some countries. How do we deal with it? What is the library source and binary size? Would they will be required or FB can work without them? How it work without them? > There are many Firebird enhancements that have been suggested: > encrypted disk images; secure client connection handshake; encrypted > backups (aka Borland Interbase); even digitally encrypted and signed > BLR routines; and native SSL engine support. A key enabler of those > features is the availability of a general set of cryptographic > functions in the engine. I would like to add an FBCrypto interface > class that provides those functions. > I don't think it should be done for v2.5, as all these benefits will not be achieved, and there are important points (questions above, plugin discussions) to resolve. > The FBCrypto class adds a C++ friendly layer over the top of the > OpenSSL libraries, and provides access to the cryptographic routines > without exposing the openssl implementation details. Is OpenSSL modular enough for we use just the crypto routines, and not the whole library including SSL sockets? > As a first step > in introducing the library, I also propose adding a number of SQL > system functions, that make use of this library to perform > encryption/hashing and general base64 encoding and decoding. > For hashing functions, I think the HASH system function should be extended with a new parameters. But to describe the return type correctly, the algorithm should be evaluated at compilation phase. > > The interface classes are described here: > > http://www.eftel.com.au/~mark.odonohue/FBCrypto.html > http://www.eftel.com.au/~mark.odonohue/ByteArray.html > Why ByteArray? We already have array classes. Also, won't the crypto functions works in chunks of bytes? > Here is a sample usage, taken from the prototype encrypt system > function implementation: > > Encryptor enc = FBCrypto::getEncryptor(); > enc.setCipher(cipherName); > enc.setPassphrase(passphrase); > enc.initEncrypt(); > ByteArray encResult = enc.encrypt(rawData); > > > And the suggested additional SQL system functions are: > > encrypt(cipherName, keyphrase, data) > decrypt(cipherName, keyphrase, data) > > createdigest(digestName, data) > verifydigest(digestName, digest, data) > > base64encode(data) > base64decode(b64string) > > hexencode(data) > hexdecode(hexstring) > > > With example usage: > > update table1 set encblob = encrypt('AES128','secret', datablob)) > > update table1 set signature = createdigest('SHA256', datablob)) > > select base64encode(encblob) from table1; > > Currently only the first two functions are implemented, although all > the underlying C++ functions exist. The encrypt function takes string > or blob data, and returns binary string or binary blob, depending on > the input data type. Does it works in chunks for blobs? > The return result for decrypt, is a binary > string or binary blob, and will need to be cast to the appropriate > type, although this could use octet strings. > I do not like the need to cast. This is another point that could be resolved in v3. Adriano |
|
From: Mark O'D. <mar...@gm...> - 2008-11-23 00:36:57
|
Hi Adriano Thanks for replying, below turned into an essay, but hopefully it answers most of your questions. I will also, in a few days, build and put up an experimental test copy, for people to download and have a look at. On Sat, Nov 22, 2008 at 11:06 PM, Adriano dos Santos Fernandes <adr...@uo...> wrote: > Mark O'Donohue wrote: >> Hi All >> > First, AFAIK strong encription is not allowed on some countries. How do > we deal with it? > Oh, you mean France :-). http://www.praxagora.com/andyo/ar/crypto_model.html I am not sure, in the old days you used to have to offer a build with encryption and without, but that doesn't seem to happen much any more, the crypto bits firefox are as strong as openssl, but they don't offers a 40 bit or no encryption download any more. But presumably to comply with this law, people in France are already building their linux boxes, and their apache webservers, from scratch, and do not use firefox, :-). And presumably they also don't use mysql which also includes openssl. http://dev.mysql.com/doc/refman/5.0/en/encryption-functions.html But I would be interested in Paul B, and Paul R's view on this,since they live in France, technically I guess they would not be able to connect to sourceforge using ssh. The other serious issue on this front is exporting from the USA, again that has relaxed over the years, I remember when all of Debian was downloadable from the US, except one file that you had to get from Finland, and technically it was ok to fax, Bruce Schneier's appendix's in Applied Cryptography , http://www.schneier.com/book-applied.html, but illegal to download them. These days, we download our redhat binaries (+openssl) without batting an eyelid. This is allowable it seems because since openssl can be obtained outside the states already, then it isn't breaking any laws. But if you build your own crypto library in the US, you can run into trouble, as Phil Zimmermann found out, with PGP, http://www.virtualschool.edu/mon/Crypto/PGPPrivacyWars.html. The acceptable outcome for everyone is rather childish and pointless, the court doesn't know how PGP escaped the US boundaries, yes it was placed on an ftp site, but they are happy that Phil wasn't personally involved in exporting it, and now they offer a crippled version for download from MIT. But, I think we should do what everyone else does, use a library that is available world wide (and openssl is the one I suggest) rather than trying to tinker or copy individual bits of code into our source base, the export boundaries and problems for US code are less, but they are not gone. > What is the library source and binary size? Would they will be required > or FB can work without them? How it work without them? > The latest source download is about 4.5meg in size. http://www.openssl.org/source/. For Linux, Unix, I would suggest linking with the OS provided openssl shared libraries, as they are part of the standard OS distribution anyway, for windows it may be preferable to link the libraries statically. The library sizes on windows are: libeay32.dll 996K ssleay32.dll 196k or statically: libeay32.lib 3,839k ssleay32.lib 600k not, sure why static ones are larger, maybe I did a debug build. I will do an experimental Firebird build, and see how it changes the binary size. But, just to add, we do get a lot of value in these openssl packages, not just the core crypto routines, but all the routines for certificate handling, pkcs7 signatures, pem encoding, even proper asn.1 parsing (asn.1 does the same mistakes as xml but they just did it earlier). > I don't think it should be done for v2.5, as all these benefits will not > be achieved, and there are important points (questions above, plugin > discussions) to resolve. > Actually, the basics I am suggesting should work fairly quickly, and be fairly stable. Interestingly, the code for database encryption is also already there, thanks to Dmitry back in 2003, I can enable his code via my routines fairly easily, the only remaining issue with Dmity's code was how to pass in the encryption key in, but that is also achievable within a reasonable timeframe. Whatever tree the crypto library is added to it should always be a conditional compilation. the FBCrypto layer I am suggesting is a good stopping point, it will perform the function or throw an exception if the implementation doesn't exist. Should it be included in 2.5 tree? I am not sure, since it would be conditional it could be done without much impact, but I suspect it all depends how long it is going to take for 2.5 to get to a final release, the release dates when you are relying on people to put in their own time for free, do tend to slip a bit. However, I must admit, which tree, was in my mind when I looked at SysFunctions.cpp, and really bit my tongue and bypassed changing this: const SysFunction* SysFunction::lookup(const Firebird::MetaName& name) { for (const SysFunction* f = functions; f->name.length() > 0; ++f) { if (f->name == name) return f; } return NULL; } to a HashMap<Key,Value> style lookup (I see Alex has added some GenericMap<Key, Value> templates) It would only take about 2hrs of refactoring, and the chance of a mistake is minimal, but isn't the sort of change you want to do prior to a release. If it goes into 3.0, then I will change this in SysFunctions.cpp, and also rework the makeXXX, evlXXX, and setParmsXXX functions as methods of classes. >> > For hashing functions, I think the HASH system function should be > extended with a new parameters. But to describe the return type > correctly, the algorithm should be evaluated at compilation phase. > I can see how that makes some sense, I haven't started on those. The current Hash is a different beast, it is like a CRC check, and not very strong from a cryptographic point of view. But I can see how changing to Hash(digestName, data) and using hash('CRC', data) or hash('SHA256', data) could work. Overall I think its probably best to leave Hash alone and create separate functions, but I don't have a strong opinion on it. > Why ByteArray? We already have array classes. Yes, good question, I really did want to use a standard existing class for that, and tried fairly hard to. I built my first version of ByteArray as an extension of Array<> but then it didn't offer good support for they type of functions I wanted, particularly reference counting, and the methods I wanted all seemed to be part of fb_string. So I looked at fb_string.h and AbstractString was based on char*, and so I tried converting that to AbstractString<charT> where it would have a common base, but it fell apart at the next level at StringComparator and StringType stage, there was also a lot of special stuff going with ensuring the \0 on the end and calls to strlen() which if mixed with unsigned chars, someone would make an easy mistake and cause trouble. I came to the conclusion that overall, a combined base, would be very messy and troublesome to maintain. It also wasn't going to do reference counting. Finally I gave up, and implemented what I wanted, although based on the functions in fb_string, and internally the private implementation class ByteArray_Impl, I suspect it will, once I get some advice on integrating Memory pools properly it probably will use Array<unsigned char>. ( I must admit I dislike Memory pools, because they cripple and make more complex what would otherwise be simple C++ code - I haven't followed the debate too closely, but those Memory Pools had better be giving good return for all the trouble they cause!). > Also, won't the crypto functions works in chunks of bytes? > Does it works in chunks for blobs? > The routines I have added works on the whole blocks of data, so it reads a whole blob, encrypts it, then writes a new blob, that is a good starting point. Internally the underlying implementation uses ssleay BIO* streams, and a source or destination BIO can be either a memory buffer (growable) or a socket. Certainly this is needed for SSL communication, and I am sure with a bit of tinkering we can create a BIO filter that works with blobs - but it is not there yet. >> > I do not like the need to cast. This is another point that could be > resolved in v3. > I agree, I don't like casting either, we need a vanilla decrypt as a starting point. But in the system function level, it is possible to do a bit more, Alex suggested adding a type-byte at the front, so the decrypt knew what type the object started life as, that is worth exploring. Hope that is enough, I will work on getting an experimental build available for download - so people can start finding bugs or problems :-). Cheers - Mark |
|
From: Adriano d. S. F. <adr...@uo...> - 2008-11-23 01:09:53
|
Mark O'Donohue wrote: > Hi Adriano > > Thanks for replying, below turned into an essay, but hopefully it > answers most of your questions. I will also, in a few days, build and > put up an experimental test copy, for people to download and have a > look at. > > On Sat, Nov 22, 2008 at 11:06 PM, Adriano dos Santos Fernandes > <adr...@uo...> wrote: > >> Mark O'Donohue wrote: >> >>> Hi All >>> >>> >> First, AFAIK strong encription is not allowed on some countries. How do >> we deal with it? >> >> > > Oh, you mean France :-). > > http://www.praxagora.com/andyo/ar/crypto_model.html > ... > But, I think we should do what everyone else does, use a library that > is available world wide (and openssl is the one I suggest) rather than > trying to tinker or copy individual bits of code into our source base, > the export boundaries and problems for US code are less, but they are > not gone. > Would be good to know how it is in Canada. >> I don't think it should be done for v2.5, as all these benefits will not >> be achieved, and there are important points (questions above, plugin >> discussions) to resolve. >> >> > > Actually, the basics I am suggesting should work fairly quickly, and > be fairly stable. Interestingly, the code for database encryption is > also already there, And I've removed it without objections in external engines branch. > Whatever tree the crypto library is added to it should always be a > conditional compilation. > But especially with static libraries, this doesn't means too much. We'll need it on the project files. > However, I must admit, which tree, was in my mind when I looked at > SysFunctions.cpp, and really bit my tongue and bypassed changing this: > > const SysFunction* SysFunction::lookup(const Firebird::MetaName& name) > { > for (const SysFunction* f = functions; f->name.length() > 0; ++f) > { > if (f->name == name) > return f; > } > > return NULL; > } > > to a HashMap<Key,Value> style lookup (I see Alex has added some > GenericMap<Key, Value> templates) Sound good. > It would only take about 2hrs of > refactoring, and the chance of a mistake is minimal, but isn't the > sort of change you want to do prior to a release. If it goes into > 3.0, then I will change this in SysFunctions.cpp, and also rework the > makeXXX, evlXXX, and setParmsXXX functions as methods of classes. > Oh no, please. Note that some makeXXX, setParamsXXX and even evlXXX are shared. Also, it doesn't sound ok for me a lot of classes (one for each function) with two or three methods. > > >> For hashing functions, I think the HASH system function should be >> extended with a new parameters. But to describe the return type >> correctly, the algorithm should be evaluated at compilation phase. >> >> > > I can see how that makes some sense, I haven't started on those. The > current Hash is a different beast, it is like a CRC check, and not > very strong from a cryptographic point of view. But I can see how > changing to Hash(digestName, data) and using hash('CRC', data) or > hash('SHA256', data) could work. Overall I think its probably best > to leave Hash alone and create separate functions, but I don't have a > strong opinion on it. > It may not be good, but is a hash like functions you are proposing. And it will need to be supported with current algorithm if an algorithm is not specified, so syntax may be HASH(<data> [, <algorithm>]). > I built my first version of ByteArray as an extension of Array<> but > then it didn't offer good support for they type of functions I wanted, > particularly reference counting > Why ref. counting? > >> Also, won't the crypto functions works in chunks of bytes? >> Does it works in chunks for blobs? >> >> > > The routines I have added works on the whole blocks of data, so it > reads a whole blob, encrypts it, then writes a new blob, that is a > good starting point. This is problematic. We current do this only when it's not possible to work in chunks. For example, in some operations with MBCS. > Internally the underlying implementation uses > ssleay BIO* streams, and a source or destination BIO can be either a > memory buffer (growable) or a socket. Certainly this is needed for > SSL communication, and I am sure with a bit of tinkering we can create > a BIO filter that works with blobs - but it is not there yet. > Never used openssl, but usage of this appears easy: http://openssl.org/docs/crypto/EVP_EncryptInit.html. > >> I do not like the need to cast. This is another point that could be >> resolved in v3. >> >> > > I agree, I don't like casting either, we need a vanilla decrypt as a > starting point. But in the system function level, it is possible to > do a bit more, Alex suggested adding a type-byte at the front, so the > decrypt knew what type the object started life as, that is worth > exploring. Not sure if we are talking about the same thing... I mean, crypt(blob) should return a blob, no problem. But crypt(string) should return a string of *some* length. And you need to calculate this length based on the input parameter, use greater value than necessary or rely on the user to put a cast(crypt(string) as varchar(XX)), and the user is not going to know this correctly either. On v3, we could have variable (dynamic) length parameters and output values. Adriano |
|
From: Mark O'D. <mar...@gm...> - 2008-11-23 11:37:36
|
Hi Adriano
I will get back to the bigger picture stuff in a bit, but in the
meantime, for nuts and bolts stuff.
But just a bit of background, previously in Firebird, I was focusing
on small picture stuff, c++ compiler errors - there were lots,
compilation dependencies, and build and install issues, putting in
readline, or whatever the bsd equivalent was, that sort of thing. I
had my hands full, and was happy to leave the engine stuff to someone
who had the time to really look at it. I certainly have an strong
aversion to changing what I don't understand.
I am glad you guys have the opportunity to learn most of the details,
and I am happy to piggyback off you to get my fairly trivial function
working right. I am certain there are still some things not quite
right yet, like the chunking, which I misunderstood, but I am equally
sure it wont take too much straightening to get right, and I
appreciate the help.
So in the vein, just a few answers and questions below.
On Sun, Nov 23, 2008 at 12:08 PM, Adriano dos Santos Fernandes
<adr...@uo...> wrote:
> Mark O'Donohue wrote:
>> Hi Adriano
>>
> This is problematic. We current do this only when it's not possible to
> work in chunks. For example, in some operations with MBCS.
>
a) Sure, I am happy to update what I have to work in chunks, sounds a good idea.
> Not sure if we are talking about the same thing... I mean, crypt(blob)
> should return a blob, no problem.
> But crypt(string) should return a string of *some* length.
b) For encrypt(x) it is easy, the return type is based on input type,
so text returns text (although unsigned char, charset none, but could
be octets), and blob returns blob.
And as you said, the return type for text is varchar : length =
len(arg[2]) + maxblocksize. The actual eval call will return a string
of up to that length.
c) In makeEncrypt to build the result I currently have :
const dsc* value = args[2];
if (value->isBlob()) {
// make blob return type
}
else (value->isText())
{
// make varchar return type len(value) + maxblocksize.
}
else {
?? now what best to do here, throw an exception?
}
d) Opps, I had a empty function for setParamsEncrypt() lets rectify that now.
setParamsEncrypt() presumably checks the input parameters, I
need parameters
1) cypher: [text,
2) key : text,
3 data : blob|text
now from other instances of setXXX all I see are:
if (argsCount >= 1 && args[0]->isUnknown())
args[0]->makeText(GUID_BODY_SIZE, ttype_ascii);
So is this only handling unknown types?, and whats best to do for the
three parameters I have (do I need to invent a max size for cypherName
and key, can I declare them as varying (there is no sample like that
in all the other functions)?, and what about arg[2] do I just check
its unknown, or can I enforce its blob|text here?
btw: one wonders if it was an ex-pascal programmer who did :
if (argsCount >= 1 && args[0]->isUnknown()) :-).
>
> On v3, we could have variable (dynamic) length parameters and output values.
>
Maybe I need to look at that, can you give a brief overview, or point
me to something ?
Cheers - Mark
|
|
From: Adriano d. S. F. <adr...@uo...> - 2008-11-23 15:11:21
|
Mark O'Donohue wrote:
> b) For encrypt(x) it is easy, the return type is based on input type,
> so text returns text (although unsigned char, charset none, but could
> be octets), and blob returns blob.
>
I think octets is more correct.
> And as you said, the return type for text is varchar : length =
> len(arg[2]) + maxblocksize. The actual eval call will return a string
> of up to that length.
>
>
> c) In makeEncrypt to build the result I currently have :
>
> const dsc* value = args[2];
>
> if (value->isBlob()) {
> // make blob return type
> }
> else (value->isText())
> {
> // make varchar return type len(value) + maxblocksize.
> }
> else {
> ?? now what best to do here, throw an exception?
> }
>
See makeRevert:
if (value->isBlob())
*result = *value;
else
result->makeVarying(value->getStringLength(), value->getTextType());
>
> d) Opps, I had a empty function for setParamsEncrypt() lets rectify that now.
>
> setParamsEncrypt() presumably checks the input parameters, I
> need parameters
> 1) cypher: [text,
> 2) key : text,
> 3 data : blob|text
>
> now from other instances of setXXX all I see are:
>
> if (argsCount >= 1 && args[0]->isUnknown())
> args[0]->makeText(GUID_BODY_SIZE, ttype_ascii);
>
> So is this only handling unknown types?, and whats best to do for the
> three parameters I have (do I need to invent a max size for cypherName
> and key, can I declare them as varying (there is no sample like that
> in all the other functions)?, and what about arg[2] do I just check
> its unknown, or can I enforce its blob|text here?
>
> btw: one wonders if it was an ex-pascal programmer who did :
> if (argsCount >= 1 && args[0]->isUnknown()) :-).
>
isUnknow() means a parameter (question mark). And this is where dynamic
varchar length for parameters go.
Current, things like this does not work: revert(?)
But this works: revert(cast(? as varchar(10)))
If this is the cast you are talking, it's ok for now.
>
>
>> On v3, we could have variable (dynamic) length parameters and output values.
>>
>>
>
> Maybe I need to look at that, can you give a brief overview, or point
> me to something ?
There is nothing proposed yet. Just some discussions of things that we
need to extend on SQLDA/VAR.
Adriano
|
|
From: Mark O'D. <mar...@gm...> - 2008-11-23 21:11:04
|
Hi Adriano
Thanks a few clarifications:
On Mon, Nov 24, 2008 at 2:10 AM, Adriano dos Santos Fernandes
<adr...@uo...> wrote:
> Mark O'Donohue wrote:
>> b) For encrypt(x) it is easy, the return type is based on input type,
>> so text returns text (although unsigned char, charset none, but could
>> be octets), and blob returns blob.
>>
> I think octets is more correct.
I agree, but my proof of concept doesnt have them yet.
>>
> See makeRevert:
> if (value->isBlob())
> *result = *value;
> else
> result->makeVarying(value->getStringLength(), value->getTextType());
>
That is, roughly, what I have,
But If I want to throw an exception during prepare rather than
execute, where is the best place to do that?
I only accept text or blob, currently I throw from evlXXX the only
other places to move that check to would seem to be the setParamsXXX
and makeXXXX functions. So to confirm, are you saying its best I
should let it other data types pass through the makeXXX and then throw
an exception in the evlEncrypt, as I do currently - eg:?
if value.isBlob()
else if value.isText()
else
throw exception
The same applies if I want to move of the excepton thrown for
"cypherName not found" to the prepare, as you suggested, where
should I move it to ? (currently it throws in evlXXXX)?
And from before:
>> I will change this in SysFunctions.cpp, and also rework the
>> makeXXX, evlXXX, and setParmsXXX functions as methods of classes.
>
>Oh no, please. Note that some makeXXX, setParamsXXX and even evlXXX are
>shared. Also, it doesn't sound ok for me a lot of classes (one for each
>function) with two or three methods.
Just curious why you don't like this, I would have thought a
structure, with a bunch of pointer to functions, and many different
implementations all created with a macro, was a classic interface and
many implementations.
True there is no performance gain, but it fits an OO structure better.
The shared methods I don't really see as a problem, probably a base
class with the core shared ones in there.
Cheers - Mark
PS: Thanks for unknown that explains what that was about :-)
|
|
From: Adriano d. S. F. <adr...@uo...> - 2008-11-23 21:36:09
|
Mark O'Donohue wrote: >> See makeRevert: >> if (value->isBlob()) >> *result = *value; >> else >> result->makeVarying(value->getStringLength(), value->getTextType()); >> >> > > That is, roughly, what I have, > > But If I want to throw an exception during prepare rather than > execute, where is the best place to do that? > In setParamsXXX or makeXXX. > I only accept text or blob, currently I throw from evlXXX the only > other places to move that check to would seem to be the setParamsXXX > and makeXXXX functions. So to confirm, are you saying its best I > should let it other data types pass through the makeXXX and then throw > an exception in the evlEncrypt, as I do currently - eg:? > If you won't want others datatypes, makeXXX is the place for verification. See makeBinShift, for example. > if value.isBlob() > else if value.isText() > else > throw exception > > The same applies if I want to move of the excepton thrown for > "cypherName not found" to the prepare, as you suggested, where > should I move it to ? (currently it throws in evlXXXX)? > evl is the right place for it. In prepare you don't have access to the string content. To have, it would be necessary to pass it as keywords, putting directly on the parser. > > And from before: > >>> I will change this in SysFunctions.cpp, and also rework the >>> makeXXX, evlXXX, and setParmsXXX functions as methods of classes. >>> >> Oh no, please. Note that some makeXXX, setParamsXXX and even evlXXX are >> shared. Also, it doesn't sound ok for me a lot of classes (one for each >> function) with two or three methods. >> > > Just curious why you don't like this, I would have thought a > structure, with a bunch of pointer to functions, and many different > implementations all created with a macro, was a classic interface and > many implementations. > > True there is no performance gain, but it fits an OO structure better. > The shared methods I don't really see as a problem, probably a base > class with the core shared ones in there. 1) Creating class declaration and implementation for each function 2) There will be needed to instantiate the classes 3) The sysfunctions doesn't store any state That's sufficient reasons to make classes there much more annoying than current implementation. Adriano |
|
From: Claudio V. C. <cv...@us...> - 2008-11-26 07:00:27
|
> -----Original Message----- > From: Adriano dos Santos Fernandes [mailto:adr...@uo...] > Sent: Sábado, 22 de Noviembre de 2008 22:09 > To: For discussion among Firebird Developers > > Mark O'Donohue wrote: > > > > Actually, the basics I am suggesting should work fairly > quickly, and > > be fairly stable. Interestingly, the code for database > encryption is > > also already there, > And I've removed it without objections in external engines branch. Not sure I understand. What did you remove in your experimental branch? The code that Dmitry protects with the macro ISC_DATABASE_ENCRYPTION? C. |
|
From: Milan B. <mi...@pa...> - 2008-11-23 11:43:43
|
Adriano dos Santos Fernandes wrote: >> I would like to add a general cryptographic library of functions for >> usage in Firebird. >> > First, AFAIK strong encription is not allowed on some countries. How do > we deal with it? Put the files for download on some server which is not in those countries. This would probably mean moving away from sourceforge.net to some other hosting solution (at least for the files). -- Milan Babuskov http://www.flamerobin.org http://www.guacosoft.com |
|
From: James M. <ja...@ma...> - 2008-11-23 21:24:06
|
Adriano dos Santos Fernandes wrote: > Mark O'Donohue wrote: > > Is OpenSSL modular enough for we use just the crypto routines, and not > the whole library including SSL sockets? > What about Botan? Or Wei Dei's Cryptlib? Doesn't OpenSSL still have the advertising clause? I think the above are handily C++ too. James |
|
From: Dmitry Y. <fir...@ya...> - 2008-11-23 07:51:51
|
Mark O'Donohue wrote: > >> I don't think it should be done for v2.5, as all these benefits will not >> be achieved, and there are important points (questions above, plugin >> discussions) to resolve. > > Actually, the basics I am suggesting should work fairly quickly, and > be fairly stable. > > Should it be included in 2.5 tree? I am not sure, since it would be > conditional it could be done without much impact, but I suspect it > all depends how long it is going to take for 2.5 to get to a final > release, the release dates when you are relying on people to put in > their own time for free, do tend to slip a bit. No way to include it into v2.5, sorry. A conditional code (turned off currently) perhaps could be allowed, but we expect to open HEAD for the v3.0 development fairly soon (before this year ends), so perhaps it would make more sense for you to wait a bit and commit this work straight there. Dmitry |
|
From: Mark O'D. <mar...@gm...> - 2008-11-23 11:42:10
|
Hi Dmitry On Sun, Nov 23, 2008 at 6:51 PM, Dmitry Yemanov <fir...@ya...> wrote: > No way to include it into v2.5, sorry. A conditional code (turned off > currently) perhaps could be allowed, but we expect to open HEAD for the > v3.0 development fairly soon (before this year ends), so perhaps it > would make more sense for you to wait a bit and commit this work > straight there. > If it went into 2.5 I would expect it to be turned off by default, but the timeframe you have for the fb3 tree becoming available sounds fine for this to be included into fb3 to me. btw: just to clarify something from earlier, without the --with-openssl specified on the ./configure line, there is no dependency created and Firebird compiles fine without openssl installed. Thanks Cheers - Mark |
|
From: Teträm C. <de...@te...> - 2008-11-23 09:20:28
|
Mark O'Donohue a écrit : > Hi Adriano > > Thanks for replying, below turned into an essay, but hopefully it > answers most of your questions. I will also, in a few days, build and > put up an experimental test copy, for people to download and have a > look at. > > On Sat, Nov 22, 2008 at 11:06 PM, Adriano dos Santos Fernandes > <adr...@uo...> wrote: > >> Mark O'Donohue wrote: >> >>> Hi All >>> >>> >> First, AFAIK strong encription is not allowed on some countries. How do >> we deal with it? >> >> > > Oh, you mean France :-). > > http://www.praxagora.com/andyo/ar/crypto_model.html > > I am not sure, in the old days you used to have to offer a build with > encryption and without, but that doesn't seem to happen much any more, > the crypto bits firefox are as strong as openssl, but they don't > offers a 40 bit or no encryption download any more. > > But presumably to comply with this law, people in France are already > building their linux boxes, and their apache webservers, from > scratch, and do not use firefox, :-). And presumably they also don't > use mysql which also includes openssl. > http://dev.mysql.com/doc/refman/5.0/en/encryption-functions.html But > I would be interested in Paul B, and Paul R's view on this,since they > live in France, technically I guess they would not be able to connect > to sourceforge using ssh. > > Well, fortunately for us, your article is on 1996 and laws changed and doesn't say anymore "no encryption" :-) Summarizing: For privacy usage, people can use a software which use encryption whiteout any worries. Only professionals must declare it: not ask for grant, only declare. And only if the software use an encryption greater than 128 bits. (It seems this limitation don't exist anymore since 2004: LEN changed it) More, if the software use a third part (ssl, etc) which already had their grants, there's no need to ask for itself. For example, PGP and GPG are already granted. So only using them doesn't imply to ask for grant. So, a simple "Worry about law" in readme should be enough.... in this case. In case of implementing a new encryption type, FB must be declared. I'm not a lawyer so my informations could be incomplete :-) Thierry |
|
From: Mark O'D. <mar...@gm...> - 2008-11-23 21:20:47
|
Hi On Sun, Nov 23, 2008 at 8:20 PM, Teträm Corp <de...@te...> wrote: > > Summarizing: > For privacy usage, people can use a software which use encryption > .... Thanks, for clarifying that, but it seems our situation is the same as most developments that use the internet and try to do so securely. Sun for instance get around export licences with a policy jar file, the strong crypto routines are all there, but if you ask for large keylength it checks the policy and throws an exception, of course you can bypass the policy file, in a variety of ways, ... or just download the unlimited policy file from the Sun website. but they can say they didnt export strong crypto from the US. and sorry about the old reference :-). Cheers - Mark |
|
From: Lester C. <le...@ls...> - 2008-11-24 07:06:21
|
Adriano dos Santos Fernandes wrote: >> I only accept text or blob, currently I throw from evlXXX the only >> other places to move that check to would seem to be the setParamsXXX >> and makeXXXX functions. So to confirm, are you saying its best I >> should let it other data types pass through the makeXXX and then throw >> an exception in the evlEncrypt, as I do currently - eg:? >> > If you won't want others datatypes, makeXXX is the place for > verification. See makeBinShift, for example. Slight aside. What effect does this have on indexing? I'm thinking in particular about ordering lists of names that are encrypted - especially once you have large number of records. -- Lester Caine - G8HFL ----------------------------- Contact - http://lsces.co.uk/lsces/wiki/?page=contact L.S.Caine Electronic Services - http://lsces.co.uk EnquirySolve - http://enquirysolve.com/ Model Engineers Digital Workshop - http://medw.co.uk// Firebird - http://www.firebirdsql.org/index.php |
|
From: Alexandre B. S. <ib...@th...> - 2008-11-24 09:04:54
|
Lester Caine wrote: > Adriano dos Santos Fernandes wrote: > >>> I only accept text or blob, currently I throw from evlXXX the only >>> other places to move that check to would seem to be the setParamsXXX >>> and makeXXXX functions. So to confirm, are you saying its best I >>> should let it other data types pass through the makeXXX and then throw >>> an exception in the evlEncrypt, as I do currently - eg:? >>> >>> >> If you won't want others datatypes, makeXXX is the place for >> verification. See makeBinShift, for example. >> > > Slight aside. > What effect does this have on indexing? > I'm thinking in particular about ordering lists of names that are > encrypted - especially once you have large number of records.\ > AFAIU a cryptographic value could be only checked for equality, inequality comparisons need the unencrypted value. For sorting purposes the values could be decrypted to memory and sorted after it see you ! -- Alexandre Benson Smith Development THOR Software e Comercial Ltda Santo Andre - Sao Paulo - Brazil www.thorsoftware.com.br |
|
From: Mark O'D. <mar...@gm...> - 2008-11-24 20:31:37
|
> > Slight aside. > What effect does this have on indexing? > I'm thinking in particular about ordering lists of names that are > encrypted - especially once you have large number of records. > Ah, Lester, noone will believe I didn't pay you to ask that question :-) it has an interesting answer. If your encryption worked, then the encrypted data looks totally random, and so there is (usually) little point trying to index it. Now, Alexandre suggested testing equality, that is true, in the old old days, you could change your Unix password to "password" then have a look at the /etc/passwd file and see if anyone else had exactly the same entry as you did, if they did, then you knew their password also had to be 'password' as well. It worked fairly well on guest accounts and the like (old Fb/Ib, before Alex's changes acted like that as well). So as a budding young hacker, one thing you could do, is build a database of SHA1/MD5 hashes containing all the words in the dictionary, and all the most common personal names, add all the special extentions of '1','2' onto the ends of the words, and all the 'E' to '3', 'o' to '0' conversions. Then you could checkout /etc/passwd maybe you had to steal it, but you could then check if any of the hashes matched those in your database if they did and then you would know that users password. Something similar was used recently by customs to prove that someone had child porn photos on their laptop, even though the customs people didn't actually see the photos on the laptop, they had just MD5'd all the files on that persons disk, and showed they were the same hashes as know child porn photos. And as it is with hashing, so it is with encryption, every encryption of the same object with the same keys gives you the same binary result, so you could tell object A was identical to object B, even though you didn't know what it was. But then along came salt, with salting you add some random bytes of data at the front or end of an object, say 20 bytes, you want know how many bytes, so you can remove them when you decrypt the object. But magically, since the 20 bytes will always be different, even if you encrypt or hash the same object many times, it will always give a completely different value, so now you have even hidden equality. This is used effectively in SSHA, invented at netscape, which is the method used for storing LDAP passwords. Now, you ask, what if we index first and then just encrypt the data record? Well it is not really a good idea, since most of the details of the object will be identifiable by analyzing the index, so you've given away what you were trying to hide. However Encrypting the whole database would work - slow but would work. And one final note :-). If you encrypt something, then try and compress it, that doesn't work very well either. Encryption tries to make the data look random, and compression, to do an effective compression, tries to find some order in the now random looking data. So if your encryption is any good, then it won't compress very much. So in practice, you should compress before you encrypt (but see note). Anyway, thanks for the chance to regurgitate, some odd little techie facts, I hope it was interesting :-) Cheers - Mark "the note": Theoretically speaking, if you compress and then encrypt, the resulting encrypted data is less random and is easier to break by brute force, than if you had encrypted just the raw data without compression - but thats the theory not what is done in practice. |
|
From: Nikolay S. <nik...@re...> - 2008-11-25 02:43:56
|
Hello, Mark! Mark O'Donohue wrote: > I would like to add a general cryptographic library of functions for > usage in Firebird. > Red Database 2.1 is based on Firebird 2.5 from this spring and has crypto plugins implemented and working. They are used for many things, among them password hashing, X.509 auth, secure tunneling for secondary auth factors (retina scans and fingerprints), table data and metadata signing, etc. You can review CryptoManager class and CryptoInterfaces.h for plugin ABI. I'm not saying that our patches for this purpose are the best, and if new and more powerful interface is available we can migrate to it, no problem. In our implementation plugin can implement one or more of the following classes of algorithms: - random number generator - hash - symmetric encryption - asymmetric encryption - digital signature calculation - the interface for certificate manipulation - cryptographic time stamp (TSP) - cryptographically strong certificate status verification (OCSP) WinCrypt plugin for Red Database is publicly available with source code. On Windows WinCrypt is natively supported. The plugin also builts on Posix systems where WinCrypt is available. It is quite popular in Russia: Linux, AIX, Solaris have a nice WinCrypt implementation made by Crypto Pro company. However the plugin interface makes no assumptions about underlying implementation. It was designed to integrate nicely with "black box" kind crypto hardware so popular in security community. It is generally illegal to use uncertified crypto code (e.g. DES, RSA, AES) in Russia. So it would be good not to include crypto libraries into Firebird directly. AFAIK, OpenSSL does not support GOST and unlike WinCrypt requires interface changes to get to work with GOST algorithms (see CryptoCom OpenSSL patches). -- Nikolay Samofatov, MBA Red Soft International +1 416 710 6854 |
|
From: Alex P. <pes...@ma...> - 2008-11-25 10:06:46
|
On Tuesday 25 November 2008 05:43, Nikolay Samofatov wrote: > Hello, Mark! > > Mark O'Donohue wrote: > > I would like to add a general cryptographic library of functions for > > usage in Firebird. > > Red Database 2.1 is based on Firebird 2.5 from this spring and has > crypto plugins implemented and working. > They are used for many things, among them password hashing, X.509 auth, > secure tunneling for secondary auth factors (retina scans and > fingerprints), table data and metadata signing, etc. > > You can review CryptoManager class and CryptoInterfaces.h for plugin ABI. > > I'm not saying that our patches for this purpose are the best, and if > new and more powerful interface is available we can migrate to it, no > problem. I'm completely agreed here with Nickolay that we need plugins implementation of crypto library. It solves very many problems, including specific laws of different countries and specific requirements of users, who are not satisfied with crypting algorythms, present in OpenSSL. What we really need in 3.0 is good general plugins interface... > WinCrypt plugin for Red Database is publicly available with source code. > On Windows WinCrypt is natively supported. It would be strange if it is not so... > The plugin also builts on Posix systems where WinCrypt is available. It > is quite popular in Russia: Linux, AIX, Solaris have a nice WinCrypt > implementation made by Crypto Pro company. I prefer to have native (OpenSSL?) implementation for mentioned platforms instead of adding dependency from one more library. > It is generally illegal to use uncertified crypto code (e.g. DES, RSA, > AES) in Russia. So it would be good not to include crypto libraries into > Firebird directly. > AFAIK, OpenSSL does not support GOST and unlike WinCrypt requires > interface changes to get to work with GOST algorithms (see CryptoCom > OpenSSL patches). If WinCrypt is really so good to implement GOST, certainly WinCrypt-based plugin may be used when GOST is required. |
|
From: Mark O'D. <mar...@gm...> - 2008-11-26 07:59:16
|
Hi Nickolay! firstly, good the hear from you! That sounds fairly impressive, I am sorry I wasn't aware you had all that, thats my fault of course, i didn't check the latest conference notes or the red database site! But I have had a quick look at the presentation you made at the conference, and downloaded the source. I certainly do not want to be re-inventing the wheel, and I am sure we can workout some way of merging our working, although yours is currently a lot more comprehensive. Now presumably you did intend for those red database changes to go into fb3, or were they scheduled for inclusion in fb2.5? Btw, I am very interested in looking at what you have done for the enhanced client logon authentication. Cheers - Mark On Tue, Nov 25, 2008 at 1:43 PM, Nikolay Samofatov <nik...@re...> wrote: > It is generally illegal to use uncertified crypto code (e.g. DES, RSA, > AES) in Russia. So it would be good not to include crypto libraries into > Firebird directly. I must admit I like things to come as standard, not to have to load a bunch of extra plugins, plugins always causes trouble, dependencies and version compatibility etc, and for bug reporting, so SSL doesnt work in fbX.Y then you know exactly what bunch of code to blame, and you dont have to clarify mixes and versions of software. , Also you don't have all the trouble of bending functionality to squeeze it through a "standard" compatible with the previous version plugin interface etc etc etc. But having said that, plugins do have a place. However, for a crypto library because in time so much depends on it, I think it would be good to have an included implementation, with the ability to plugin a different one for special circumstances, such as Nickolay's, where for them AES is still considered uncertified for use. Cheers - Mark BTW, I cant seem to access http:://www.wincrypt.com it seems to not return anything, is that the best site for wincrypt? |
|
From: Alex P. <pes...@ma...> - 2008-11-26 13:48:57
|
On Wednesday 26 November 2008 10:59, Mark O'Donohue wrote: > Now presumably you did intend for > those red database changes to go into fb3, or were they scheduled for > inclusion in fb2.5? 3.0, not 2.5 > I must admit I like things to come as standard, not to have to load a > bunch of extra plugins, plugins always causes trouble, dependencies > and version compatibility etc, and for bug reporting, so SSL doesnt > work in fbX.Y then you know exactly what bunch of code to blame, and > you dont have to clarify mixes and versions of software. , Also you > don't have all the trouble of bending functionality to squeeze it > through a "standard" compatible with the previous version plugin > interface etc etc etc. Starting with 3.0 firebird is planned to become OSRI-compliant, becoming itself a set of 'plugins' - redirector is plugin, engine is plugin.Therefore we should anyway have to solve most of mentioned problems. Particularly, old engine should be plaggable into newer yValve. I.e. we must have (and have in vulcan) ability to extend plugin's interface in newer versions. And we will anyway need to take into an account when analyzing bug reports that problems might happen in old engine. I.e. I do not see great difference, should we take into an account only other version of an engine or also other version of crypto-plugin. > But having said that, plugins do have a place. However, for a crypto > library because in time so much depends on it, I think it would be > good to have an included implementation, OK, suppose we use OpenSSL as included implementaion. On most of *nix systems (but not on all!) required support is always present. But - who knows excatly what version of dynamic library is used, and what vendor-specific patches were applied? (The last factor is specially interesting - I have some samples how firebird was broken when badly merged in some linux distros and freebsd.) I.e. we do not solve a problem - what "SSL doesnt work in fbX.Y". This problem is not solved by use of crypto-plugins, but you see yourself - it does not go away with use of If we use static linkage (like you suggest for windows) things look even worse. First, in case of security problem found in OpenSSL we need to rebuild firebird and quickly release fresh version. BTW, quickly often happens to be months in our case... Second, from where do you suggest to take source code of OpenSSL when building firebird for windows? Merge it into the tree? I completely dislike such solution. We already have enough troubles with icu in our tree, I don't want any more. Suggesting people to download OpenSSL and setup it before building firebird is better, but I already hear people crying that 'building firebird became too complex'. With all crypto-support, implemenyted as plugins, both problems are gone - on windows we use by default windows crypto API based plugin, and have absolutely no troubles of this two kinds. Alex. |
|
From: Nikolay S. <nik...@re...> - 2008-12-04 02:22:32
|
Hello, Mark! Nice to hear from you as well. I'm still busy fighting fires, and cannot read/write fb-devel too often. I cannot take much credit for this security work. I only made initial top-level designs and reviewed the specs iteratively. The specs were written by Roman Simakov and his Murom team. The detailed specs sign-off was done by russian database security engineers who do the similar work for MySQL, Oracle and MS SQL. I believe they prefer not to have their names called in public too often. :-) I almost never touched the implementation so 90%+ of credits should go to Murom guys who spent 10+ man-years on the thing by now. Mark O'Donohue wrote: > Now presumably you did intend for > those red database changes to go into fb3, or were they scheduled for > inclusion in fb2.5? > FB3 or whenever the stars align. Merge requires resources, and this is a problem. > Btw, I am very interested in looking at what you have done for the > enhanced client logon authentication. > You can get in touch with Roman, he has all the details. > BTW, I cant seem to access http:://www.wincrypt.com it seems to not > return anything, > is that the best site for wincrypt? > You can read about it on MSDN, e.g. http://msdn.microsoft.com/en-us/library/aa388162(VS.85).aspx We call it WinCrypt because the corresponding include header file is called "wincrypt.h". I think official name now is Microsoft CryptoAPI, but I also heard about CAPI and other names for the same thing. Although it is designed for Windows, there are implementations for other platforms as well (the story similar to ODBC). Yes, OpenSSL use/support would be great, but we didn't work on that. The plugins are almost a must, because real security guys use special hardware with "Classified" API. Think about it - not only the implementation is secret, but API and guidelines for its use are also a secret. For your engine to work with this HW you have to hire guys sitting at undisclosed location, and let them write crypto-plugin for your interface and distribute/support it. It is a way easier and cheaper if there is no OpenSSL in the middle, and they write to defined plug-in spec. Note that every line of code has to go through static and dynamic flow analysis, so adding big extra packages (like OpenSSL) into sensitive places costs money to certify, lots of money. We may discuss more details privately with ICQ 114101470 or Skype nikolay.samofatov, if you want. -- Nikolay Samofatov, MBA Red Soft International +1 416 710 6854 |
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X