[Libvncserver-common] libgcrypt (still) crashing when connecting to Mac hosts

[Kevin F. <kf...@kd...>]

Heya,

this has been already discussed at the beginning of 2014:
  http://sourceforge.net/p/libvncserver/mailman/message/32165820/

This is *exactly* the same issue, but I'm resending this in hope to get this 
fixed. KRDC is unusable for me now.

I'm also getting a crash when using KRDC to connect to a Mac host (running OSX 
10.7, if that helps). Backtrace is attached, see [1]

Error message is:

Fatal: failed to create the RNG lock: Invalid argument
FATAL: failed to acquire the FSM lock in libgrypt: Invalid argument

libgcrypt version is: 1.6.1-2ubuntu1

I also tried compiling libvncserver with SSL support and without gcrypt 
support but compilation fails in this mode:

../libvncserver/.libs/libvncserver.so: error: undefined reference to 
'gcry_md_open'
../libvncserver/.libs/libvncserver.so: error: undefined reference to 
'gcry_md_write'
../libvncserver/.libs/libvncserver.so: error: undefined reference to 
'gcry_md_ctl'
../libvncserver/.libs/libvncserver.so: error: undefined reference to 
'gcry_md_get_algo_dlen'
../libvncserver/.libs/libvncserver.so: error: undefined reference to 
'gcry_md_read'
collect2: error: ld returned 1 exit status

Note: I have no real clue about VNC/libvncserver/libgcrypt altogether, but 
this seems like a real issue people are having which has been around for more 
than a year now. Let's get this fixed, happy to help.

Greets

[1] Backtrace:
(gdb) bt
#0  0x00007ffff3e41d27 in __GI_raise (sig=sig@entry=6) at 
../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007ffff3e43418 in __GI_abort () at abort.c:89
#2  0x00007fffe391e8f4 in lock_fsm () at fips.c:235
#3  0x00007fffe391ec01 in fips_new_state (new_state=STATE_FATALERROR) at 
fips.c:759
#4  0x00007fffe391f0b8 in _gcry_fips_signal_error (srcfile=0x7fffe39b49f0 
"misc.c", srcline=140, srcfunc=0x7fffe39b4b00 <__FUNCTION__.7625> 
"_gcry_logv", is_fatal=1, description=0x7fffe39b4a3b "internal
 error (fatal or bug)") at fips.c:732
#5  0x00007fffe3917efa in _gcry_logv (level=level@entry=40, 
fmt=fmt@entry=0x7fffe39d24e8 "failed to create the RNG lock: %s\n", 
arg_ptr=arg_ptr@entry=0x7fffe0bd68f8) at misc.c:140
#6  0x00007fffe3918274 in _gcry_log_fatal (fmt=fmt@entry=0x7fffe39d24e8 
"failed to create the RNG lock: %s\n") at misc.c:230
#7  0x00007fffe39a9665 in basic_initialization () at random-fips.c:201
#8  0x00007fffe39aa2e1 in _gcry_rngfips_initialize (full=1) at random-
fips.c:746
#9  _gcry_rngfips_randomize (buffer=0x7fffd4003a90, length=16, 
level=GCRY_STRONG_RANDOM) at random-fips.c:835
#10 0x00007fffe39a7c80 in _gcry_random_bytes (nbytes=nbytes@entry=16, 
level=GCRY_STRONG_RANDOM) at random.c:324
#11 0x00007fffe39b23d7 in _gcry_mpi_randomize (w=0x7fffd4003a50, 
nbits=<optimized out>, level=<optimized out>) at mpiutil.c:586
#12 0x00007fffe3bff25c in InitialiseRFBConnection () from /usr/lib/x86_64-
linux-gnu/libvncclient.so.0
#13 0x00007fffe3c01064 in rfbInitClient () from /usr/lib/x86_64-linux-
gnu/libvncclient.so.0
#14 0x00007fffe3e17e84 in VncClientThread::clientCreate 
(this=this@entry=0xb25220, reinitialising=reinitialising@entry=false) at 
../../vnc/vncclientthread.cpp:555
#15 0x00007fffe3e182b0 in VncClientThread::run (this=0xb25220) at 
../../vnc/vncclientthread.cpp:463
#16 0x00007ffff5fe239f in ?? () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
#17 0x00007ffff23320a5 in start_thread (arg=0x7fffe0bd7700) at 
pthread_create.c:309
#18 0x00007ffff3f0577d in clone () at 
../sysdeps/unix/sysv/linux/x86_64/clone.S:111


-- 
Kevin Funk | kf...@kd... | http://kfunk.org