Re: [Opensc-devel] ECDSA, engine_pkcs11, libp11 and OpenSSL

[Markus K. <ko...@rr...>]

On 09/21/2013 11:45 PM, Markus Kötter wrote:
> And I even tried to do it, n with ECDSA but fixing the memory leaks of
> the RSA code.

I forked and created a branches.
https://github.com/commonism/libp11/commits/memoryleaks
https://github.com/commonism/engine_pkcs11/commits/memoryleaks


libp11 leaking the ERR strings when unloading the engine:

https://github.com/commonism/libp11/commit/46dacbe8f5badde89d25289faab82232311822b4
https://github.com/commonism/engine_pkcs11/commit/67244a1cef3decc5b896be5adb9dd771262ab37a

engine_pkcs11 free's the mallocs claimed by PKCS11_ structures.
https://github.com/commonism/engine_pkcs11/commit/fbae0727e88fd20e1cba6ec60799dc4fe705cf97

> Problems ...
>
> idx 0 - which is returned by default, is used by p11 already as app_data.

I have a funny workaround in place.

> ex data slots claimed with RSA_get_ex_new_index last forever.
> Unload the engine, there is no way to drop the slot, so the callback
> will be called, if the memory is unmapped as the engine is unloaded, you
> are lost.
> Reload the engine, get mapped in the same location, you get the
> callbacks for the unmapped engines as well.

Not fixable.
"Therefore - better not un-load the engine."
Currently this is broken for all dynamic engines.

> The API exposed by OpenSSLs ex_data.c is not sufficient to remove the
> slot manually.

Once OpenSSL comes up with a way to get rid of a claimed slot, I will 
update this.

> Calling PKCS11_release_all_slots with the required arguments in the ex
> data destructor of an EVP_PKEY/RSA results in recursion, as
> PKCS11_release_all_slots calls EVP_PKEY_free as well.

Fixed, remembering the claimed PKCS11_KEY and setting the evp_key NULL 
before destroying the rest prevents the recursion easily.


If we can agree this is a proper approach to free the claimed memory, 
I'll make this work for certs as well, currently only keys are taken 
care of.


MfG
Markus