[Flightgear-devel] Threaded Nasal GC - simgear ctest failures on Linux.

[Edward d'A. <tru...@gm...>]

Hi Richard,

I've been looking at how we can fix the simgear ctests that have been
failing on Linux since your threaded Nasal GC work was merged [1].
This seems to be causing blockage issues in the Jenkins testing
pipelines.

Due to the large number of 'git bisect skip' commands required due to
Linux compile issues, bisecting is simply not possible [2].  So
instead I took a pot shot and reverted your commit from June 11:

92a3c8bbd823ce2aee5cbe434cc0ea5c03c47f71 Added Nasal garbage
collection background thread

That is the commit that causes the tests to fail on Linux.  Via gdb,
it seems like there are some locking issues [3, 4].  Any idea what
this might be about?  I don't see such issues when running fgfs but I
wonder if this segfault could affect the binary.

Cheers,

Edward


[1]  The failing ctests:

"""
      Start 27: nasal_gc_test-boost_test
27/48 Test #27: nasal_gc_test-boost_test ............***Exception:
SegFault  0.01 sec
      Start 28: nasal_num-boost_test
28/48 Test #28: nasal_num-boost_test ................***Exception:
SegFault  0.01 sec
"""


[2]  My git bisect results:

"""
There are only 'skip'ped commits left to test.
The first bad commit could be any of:
b22a8debf27cf470d091419a279a436cfa09f507
01ac9a71b77e4405d0dbdbd3aa822269870c9d2a
92a3c8bbd823ce2aee5cbe434cc0ea5c03c47f71
ab8b1d39bdc4b47eb3e5a00a186a2c4b7f27f29b
c71f2874983b04d42136d9a2e779819dc0945d7b
098c5b0924221d2047dd35f9cd1fefc4b157ce0e
b7f8fbe7a05fb7bd8819d78977a58f463e11b865
2cbbb29eb534e634dd42049399e504b1e7aa8389
7a903361e6bdc161c93167f64996333a05b0ff4e
c0677ad8c5b90bbd0c349edb6b01e26833f251be
798b90e0a58c3444275f139e74bfe62a50259dc2
3d841a214c9057e3f94c6fa9e6fba07b467f000d
cba902f22b9b2467c71e58fb2c7e87e5e6d38bd9
3b3093c72eaf3cacbb0a1411b9551706e52009bf
8eb51e813fd62098ba5c06c4746759187a8984f4
d8a46cffa5198260e980b93ffb4899665b65553d
We cannot bisect more!
"""


[3]  Debugging the test-simgear_nasal_cppbind-nasal_gc_test binary:

[edward@localhost flightgear-simgear (next)]$
  BRANCH: next  (fa754eca) / 6 uncommitted changes
$ gdb /flightgear/src/flightgear-simgear/build_debug/simgear/nasal/cppbind/test-simgear_nasal_cppbind-nasal_gc_test
GNU gdb (GDB) 7.12-16.1.mga6 (Mageia release 6)

[snip]

Reading symbols from
/flightgear/src/flightgear-simgear/build_debug/simgear/nasal/cppbind/test-simgear_nasal_cppbind-nasal_gc_test...done.
(gdb) run
Starting program:
/flightgear/src/flightgear-simgear/build_debug/simgear/nasal/cppbind/test-simgear_nasal_cppbind-nasal_gc_test
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib64/libthread_db.so.1".

[snip]

Program received signal SIGSEGV, Segmentation fault.
__GI___pthread_mutex_lock (mutex=0x0) at ../nptl/pthread_mutex_lock.c:68
68        unsigned int type = PTHREAD_MUTEX_TYPE_ELISION (mutex);
Missing separate debuginfos, use: debuginfo-install
lib64krb53-1.15.1-2.4.mga6.x86_64
lib64openssl1.0.0-1.0.2r-1.mga6.x86_64
lib64ssh2_1-1.7.0-2.1.mga6.x86_64
(gdb) bt
#0  __GI___pthread_mutex_lock (mutex=0x0) at ../nptl/pthread_mutex_lock.c:68
#1  0x00000000004610c2 in SGMutex::PrivateData::lock (this=0x0) at
/flightgear/src/flightgear-simgear/simgear/threads/SGThread.cxx:256
#2  0x0000000000460223 in SGMutex::lock (this=0x4c5b78
<global_logStreamLock>) at
/flightgear/src/flightgear-simgear/simgear/threads/SGThread.cxx:377
#3  0x0000000000489498 in SGGuard<SGMutex>::SGGuard
(this=0x7fffffffd1c8, l=...) at
/flightgear/src/flightgear-simgear/simgear/threads/SGGuard.hxx:18
#4  0x0000000000487802 in sglog () at
/flightgear/src/flightgear-simgear/simgear/debug/logstream.cxx:723
#5  0x000000000045cdcb in
nasal::NasalMainLoopRecipient::NasalMainLoopRecipient (this=0x4c5a00
<nasal::mrl>) at
/flightgear/src/flightgear-simgear/simgear/nasal/cppbind/NasalEmesaryInterface.hxx:70
#6  0x000000000045c079 in __static_initialization_and_destruction_0
(__initialize_p=1, __priority=65535) at
/flightgear/src/flightgear-simgear/simgear/nasal/cppbind/detail/to_nasal_helper.cxx:42
#7  0x000000000045c0d6 in _GLOBAL__sub_I_to_nasal_helper.cxx(void) ()
at /flightgear/src/flightgear-simgear/simgear/nasal/cppbind/detail/to_nasal_helper.cxx:140
#8  0x00000000004a0ced in __libc_csu_init (argc=argc@entry=1,
argv=argv@entry=0x7fffffffd528, envp=0x7fffffffd538) at elf-init.c:88
#9  0x00007ffff40c658f in __libc_start_main (main=0x4410bd <main(int,
char**)>, argc=1, argv=0x7fffffffd528, init=0x4a0ca0
<__libc_csu_init>, fini=<optimized out>, rtld_fini=<optimized out>,
stack_end=0x7fffffffd518) at libc-start.c:245
#10 0x0000000000440f69 in _start () at ../sysdeps/x86_64/start.S:118


[4]  Debugging the test-simgear_nasal_cppbind-nasal_num binary:

[edward@localhost flightgear-simgear (next)]$
  BRANCH: next  (fa754eca) / 6 uncommitted changes
$ gdb /flightgear/src/flightgear-simgear/build_debug/simgear/nasal/cppbind/test-simgear_nasal_cppbind-nasal_num
GNU gdb (GDB) 7.12-16.1.mga6 (Mageia release 6)

[snip]

Reading symbols from
/flightgear/src/flightgear-simgear/build_debug/simgear/nasal/cppbind/test-simgear_nasal_cppbind-nasal_num...done.
(gdb) run
Starting program:
/flightgear/src/flightgear-simgear/build_debug/simgear/nasal/cppbind/test-simgear_nasal_cppbind-nasal_num
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib64/libthread_db.so.1".

[snip]

Program received signal SIGSEGV, Segmentation fault.
__GI___pthread_mutex_lock (mutex=0x0) at ../nptl/pthread_mutex_lock.c:68
68        unsigned int type = PTHREAD_MUTEX_TYPE_ELISION (mutex);
Missing separate debuginfos, use: debuginfo-install
lib64krb53-1.15.1-2.4.mga6.x86_64
lib64openssl1.0.0-1.0.2r-1.mga6.x86_64
lib64ssh2_1-1.7.0-2.1.mga6.x86_64
(gdb) bt
#0  __GI___pthread_mutex_lock (mutex=0x0) at ../nptl/pthread_mutex_lock.c:68
#1  0x0000000000460f64 in SGMutex::PrivateData::lock (this=0x0) at
/flightgear/src/flightgear-simgear/simgear/threads/SGThread.cxx:256
#2  0x00000000004600c5 in SGMutex::lock (this=0x4c5ad8
<global_logStreamLock>) at
/flightgear/src/flightgear-simgear/simgear/threads/SGThread.cxx:377
#3  0x00000000004892e2 in SGGuard<SGMutex>::SGGuard
(this=0x7fffffffd1c8, l=...) at
/flightgear/src/flightgear-simgear/simgear/threads/SGGuard.hxx:18
#4  0x00000000004876a4 in sglog () at
/flightgear/src/flightgear-simgear/simgear/debug/logstream.cxx:723
#5  0x000000000045cc6d in
nasal::NasalMainLoopRecipient::NasalMainLoopRecipient (this=0x4c5960
<nasal::mrl>) at
/flightgear/src/flightgear-simgear/simgear/nasal/cppbind/NasalEmesaryInterface.hxx:70
#6  0x000000000045bf1b in __static_initialization_and_destruction_0
(__initialize_p=1, __priority=65535) at
/flightgear/src/flightgear-simgear/simgear/nasal/cppbind/detail/to_nasal_helper.cxx:42
#7  0x000000000045bf78 in _GLOBAL__sub_I_to_nasal_helper.cxx(void) ()
at /flightgear/src/flightgear-simgear/simgear/nasal/cppbind/detail/to_nasal_helper.cxx:140
#8  0x00000000004a0a5d in __libc_csu_init (argc=argc@entry=1,
argv=argv@entry=0x7fffffffd528, envp=0x7fffffffd538) at elf-init.c:88
#9  0x00007ffff40c658f in __libc_start_main (main=0x44007d <main(int,
char**)>, argc=1, argv=0x7fffffffd528, init=0x4a0a10
<__libc_csu_init>, fini=<optimized out>, rtld_fini=<optimized out>,
stack_end=0x7fffffffd518) at libc-start.c:245
#10 0x000000000043ff29 in _start () at ../sysdeps/x86_64/start.S:118
(gdb)