[Opensc-devel] Double locking/double initialization deadlock after being called by ENGINE_init?

[Rico H. <ri...@gm...>]

Hi,

I have a problem when trying to use the Feitian ePass2003 in combination
with OpenSC 0.13.0, in mod_ssl.

Now, mod_ssl cannot actually use the PKCS11 engine by itself, but I found a
patch that properly seems to load the engine here:
https://issues.apache.org/bugzilla/show_bug.cgi?id=52473 (also posted this
issue there)

After applying the patch, the PKCS11 engine seems to be loaded correctly,
but deadlocks during its 2nd initialization (mod_ssl does 2
initializations, one to check the config and then one to actually start up.
It does clean up in between.).

It seems to be a non-reentrant double-locking problem (see stack trace:
ENGINE_init (#23) acquires a lock, but then later in engine_table_select
(#7) the thread tries to acquire the same lock again and deadlocks), but
then I have no problem why it doesn't deadlock on the first call to
ENGINE_init().

I don't know enough about the internals of OpenSSL to further comment on
what library is at fault here, so I hope you guys can figure that out.

I would post this as a bug on Trac but that won't let me log in so I'm
posting it here, hope that's okay.

Best regards,
Rico

-----

stack trace:

#0  0xb776c424 in __kernel_vsyscall ()
#1  0xb76bb4d2 in __lll_lock_wait () from
/lib/i386-linux-gnu/libpthread.so.0
#2  0xb76b6ed4 in _L_lock_776 () from /lib/i386-linux-gnu/libpthread.so.0
#3  0xb76b6d13 in pthread_mutex_lock () from
/lib/i386-linux-gnu/libpthread.so.0
#4  0xb76e07d0 in apr_thread_mutex_lock () from /usr/lib/libapr-1.so.0
#5  0xb7380617 in ssl_util_thr_lock () from
/usr/lib/apache2/modules/mod_ssl.so
#6  0xb718f9f5 in CRYPTO_lock (mode=mode@entry=9, type=type@entry=30,
file=file@entry=0xb72a73f4 "eng_table.c", line=line@entry=258) at
cryptlib.c:604
#7  0xb71fc077 in engine_table_select (table=table@entry=0xb72fe8d4
<cipher_table>, nid=nid@entry=418) at eng_table.c:258
#8  0xb71fdb55 in ENGINE_get_cipher_engine (nid=418) at tb_cipher.c:115
#9  0xb72118c2 in EVP_CipherInit_ex (ctx=ctx@entry=0xbf89b7c0,
cipher=cipher@entry=0xb72ef9e0 <aes_128_ecb>, impl=impl@entry=0x0,
key=key@entry=0xb6e60068
"\001\002\003\004\005\006\a\b\t\n\v\f\r\016\017\020\001\002\003\004\005\006\a\b\t\n\v\f\r\016\017\020\277\303)\021\307\030\303@",
iv=iv@entry=0xbf89b84c "", enc=enc@entry=1) at evp_enc.c:147
#10 0xb7211a33 in EVP_EncryptInit_ex (ctx=0xbf89b7c0, cipher=0xb72ef9e0
<aes_128_ecb>, impl=0x0, key=0xb6e60068
"\001\002\003\004\005\006\a\b\t\n\v\f\r\016\017\020\001\002\003\004\005\006\a\b\t\n\v\f\r\016\017\020\277\303)\021\307\030\303@",
iv=0xbf89b84c "") at evp_enc.c:292
#11 0xb6d63aa3 in ?? () from /usr/lib/libopensc.so.3
#12 0xb6d65aa5 in ?? () from /usr/lib/libopensc.so.3
#13 0xb6d66d05 in ?? () from /usr/lib/libopensc.so.3
#14 0xb6d670f7 in ?? () from /usr/lib/libopensc.so.3
#15 0xb6d6729f in ?? () from /usr/lib/libopensc.so.3
#16 0xb6d01c85 in sc_connect_card () from /usr/lib/libopensc.so.3
#17 0xb6fcdadd in ?? () from /usr/lib/opensc-pkcs11.so
#18 0xb6fce076 in ?? () from /usr/lib/opensc-pkcs11.so
#19 0xb6fc9521 in ?? () from /usr/lib/opensc-pkcs11.so
#20 0xb703d973 in PKCS11_CTX_load () from
/usr/lib/i386-linux-gnu/libp11.so.2
#21 0xb704cdde in ?? () from /usr/lib/engines/engine_pkcs11.so
#22 0xb71fae8b in engine_unlocked_init (e=e@entry=0xb8c383f8) at
eng_init.c:67
#23 0xb71fb000 in ENGINE_init (e=0xb8c383f8) at eng_init.c:130
#24 0xb736a68a in ssl_ossle_get_engine () from
/usr/lib/apache2/modules/mod_ssl.so
#25 0xb736b03f in ssl_init_Engine () from
/usr/lib/apache2/modules/mod_ssl.so
#26 0xb736ae1d in ssl_init_Module () from
/usr/lib/apache2/modules/mod_ssl.so
#27 0xb77c51ee in ap_run_post_config (pconf=pconf@entry=0xb7492018,
plog=0xb7460018, ptemp=0xb745e018, s=s@entry=0xb748c6a8) at config.c:95
#28 0xb77ae587 in main (argc=3, argv=0xbf89c464) at main.c:688