Segfault when running step 3
Brought to you by:
hld
Menu
▾
▴
#7 Segfault when running step 3
Using v3.0031 (and previous versions), IBDLD crashes with a segfault when I don't specify steps on the command line, but doesn't crash when I run each step in order. Some snippets from GDB follow:
**-- BEGIN --**
Run IBD estimation for chromosome 21 !
There are 126253 pairs in Chromosome 21 to be analyzed in total!
Begin to compute IBD probabilities for Chromosome 21 !
The Pair 0 IBD comoutation use thread 2 out of 12
Compute pair: 100
Compute pair: 200
*** glibc detected *** /home/gringer/install/ibdld/IBDLDv3.0031/IBDLD: free(): invalid pointer: 0x00007fffc4752b88 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x76d76)[0x7ffff685ed76]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x6c)[0x7ffff6863aac]
/home/gringer/install/ibdld/IBDLDv3.0031/IBDLD[0x413b9a]
/home/gringer/install/ibdld/IBDLDv3.0031/IBDLD[0x414db2]
/home/gringer/install/ibdld/IBDLDv3.0031/IBDLD[0x4224c3]
/home/gringer/install/ibdld/IBDLDv3.0031/IBDLD[0x48136a]
/usr/lib/x86_64-linux-gnu/libgomp.so.1(+0x906a)[0x7ffff6fae06a]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50)[0x7ffff6b78b50]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7ffff68c2a7d]
======= Memory map: ========
00400000-004bc000 r-xp 00000000 08:02 4295806622 /home/gringer/install/ibdld/IBDLDv3.0031/IBDLD
006bc000-006bd000 rw-p 000bc000 08:02 4295806622 /home/gringer/install/ibdld/IBDLDv3.0031/IBDLD
006bd000-033bc000 rw-p 00000000 00:00 0 [heap]
7fffb0000000-7fffb0860000 rw-p 00000000 00:00 0
7fffb0860000-7fffb4000000 ---p 00000000 00:00 0
7fffb4000000-7fffb4733000 rw-p 00000000 00:00 0
7fffb4733000-7fffb8000000 ---p 00000000 00:00 0
...
Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffee72a700 (LWP 17926)]
0x00007ffff681a475 in *__GI_raise (sig=<optimized out>) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
64 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0 0x00007ffff681a475 in *__GI_raise (sig=<optimized out>) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1 0x00007ffff681d6f0 in *__GI_abort () at abort.c:92
#2 0x00007ffff685552b in __libc_message (do_abort=<optimized out>, fmt=<optimized out>) at ../sysdeps/unix/sysv/linux/libc_fatal.c:189
#3 0x00007ffff685ed76 in malloc_printerr (action=3, str=0x7ffff6933b9c "free(): invalid pointer", ptr=<optimized out>) at malloc.c:6283
#4 0x00007ffff6863aac in *__GI___libc_free (mem=<optimized out>) at malloc.c:3738
#5 0x0000000000413b9a in _M_dispose (__a=..., this=0x7fffc4752b88) at /usr/include/c++/4.7/bits/basic_string.h:246
#6 ~basic_string (this=0x7fffc474c0e0, __in_chrg=<optimized out>) at /usr/include/c++/4.7/bits/basic_string.h:536
#7 _Destroy<std::basic_string<char> > (__pointer=0x7fffc474c0e0) at /usr/include/c++/4.7/bits/stl_construct.h:95
#8 __destroy<std::basic_string<char>*> (__last=0x7fffc474c100, __first=0x7fffc474c0e0) at /usr/include/c++/4.7/bits/stl_construct.h:105
#9 _Destroy<std::basic_string<char>*> (__last=0x7fffc474c100, __first=<optimized out>) at /usr/include/c++/4.7/bits/stl_construct.h:128
#10 _Destroy<std::basic_string<char>*, std::basic_string<char> > (__last=0x7fffc474c100, __first=<optimized out>) at /usr/include/c++/4.7/bits/stl_construct.h:155
#11 ~vector (this=0x7fffc474f9f0, __in_chrg=<optimized out>) at /usr/include/c++/4.7/bits/stl_vector.h:403
#12 _Destroy<std::vector<std::basic_string<char> > > (__pointer=0x7fffc474f9f0) at /usr/include/c++/4.7/bits/stl_construct.h:95
#13 std::_Destroy_aux<false>::__destroy<std::vector<std::string, std::allocator<std::string> >*> (__first=0x7fffc474f9f0, __last=0x7fffc47511f0) at /usr/include/c++/4.7/bits/stl_construct.h:105
#14 0x0000000000414db2 in _Destroy<std::vector<std::basic_string<char> >*> (__last=<optimized out>, __first=<optimized out>) at /usr/include/c++/4.7/bits/stl_construct.h:128
#15 _Destroy<std::vector<std::basic_string<char> >*, std::vector<std::basic_string<char> > > (__last=<optimized out>, __first=<optimized out>) at /usr/include/c++/4.7/bits/stl_construct.h:155
#16 std::vector<std::vector<std::string, std::allocator<std::string> >, std::allocator<std::vector<std::string, std::allocator<std::string> > > >::_M_insert_aux (this=0x7fffffffc9a0, __position=...,
__x=...) at /usr/include/c++/4.7/bits/vector.tcc:382
#17 0x00000000004224c3 in push_back (__x=..., this=0x7fffffffc9a0) at /usr/include/c++/4.7/bits/stl_vector.h:893
#18 KinshipIBC_Construct (MultiplyRowNameListKin=..., MultiplyRowNameListIBC=..., MultiplyRowKinship=..., MultiplyIBC=..., SNP_NumEachPairInChrKin=..., SNP_NumEachPairInChrIBC=..., Alpha_BetaProb9=...,
Alpha_BetaProb2=..., iPairList=..., sStudyPairNameList=..., iPair=@0x7fffee729890: 257, ibcCheck=@0x7fffffffd590: 0) at ./IBDFold/CommonInformTreat.cpp:846
#19 0x000000000048136a in MultiplyMethodsComputaionResultOutput(std::string, std::string, std::string, std::string, std::string, int, int, short, int, int, float, int, int, bool const&, bool const&, float, float, int, int, long, double, int, int, std::string&, float, std::string, unsigned long&, int const&) [clone ._omp_fn.0] () at ./IBDFold/TriCoreProgram.cpp:2140
#20 0x00007ffff6fae06a in ?? () from /usr/lib/x86_64-linux-gnu/libgomp.so.1
#21 0x00007ffff6b78b50 in start_thread (arg=<optimized out>) at pthread_create.c:304
#22 0x00007ffff68c2a7d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#23 0x0000000000000000 in ?? ()
**-- END --**
The backtrace indicates that the last IBDLD-specific code point was this:
#18 KinshipIBC_Construct (MultiplyRowNameListKin=..., MultiplyRowNameListIBC=..., MultiplyRowKinship=..., MultiplyIBC=..., SNP_NumEachPairInChrKin=..., SNP_NumEachPairInChrIBC=..., Alpha_BetaProb9=...,
Alpha_BetaProb2=..., iPairList=..., sStudyPairNameList=..., iPair=@0x7fffee729890: 257, ibcCheck=@0x7fffffffd590: 0) at ./IBDFold/CommonInformTreat.cpp:846
Line 846 of CommonInformTreat.cpp is a push_back onto a vector:
MultiplyRowNameListKin.push_back(sStudyPairNameList[iPair]);
So either 'sStudyPairNameList[iPair]' is the problem (or a downstream effect), or the vector, 'MultiplyRowNameListKin' is the problem (or a downstream effect).
All of these point to a problem further back in the code, because all are passed by reference as function arguments. Both 'sStudyPairNameList' and 'iPair' are const and passed by reference, and 'MultiplyRowNameListKin' is passed by reference (and can be modified).
Discussion
Actually, this bug is happening now (v3.0031) when just doing step 3 alone:
run 1:
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff48af700 (LWP 18445)]
malloc_consolidate (av=0x7fffe4000020) at malloc.c:5155
5155 malloc.c: No such file or directory.
(gdb) bt
#0 malloc_consolidate (av=0x7fffe4000020) at malloc.c:5155
#1 0x00007ffff68606a8 in _int_free (av=0x7fffe4000020, p=0x7fffe4782d20) at malloc.c:5034
#2 0x00007ffff6863aac in *__GI___libc_free (mem=<optimized out>) at malloc.c:3738
#3 0x00000000004926a3 in aligned_free (ptr=<optimized out>) at ./IBDFold/Eigen/src/Core/util/Memory.h:225
#4 conditional_aligned_free<true> (ptr=<optimized out>) at ./IBDFold/Eigen/src/Core/util/Memory.h:299
#5 conditional_aligned_delete_auto<float, true> (ptr=<optimized out>, size=<optimized out>) at ./IBDFold/Eigen/src/Core/util/Memory.h:425
#6 ~DenseStorage (this=0x7ffff48ae730, __in_chrg=<optimized out>) at ./IBDFold/Eigen/src/Core/DenseStorage.h:275
#7 ~PlainObjectBase (this=0x7ffff48ae730, __in_chrg=<optimized out>) at ./IBDFold/Eigen/src/Core/PlainObjectBase.h:72
#8 ~Array (this=0x7ffff48ae730, __in_chrg=<optimized out>) at ./IBDFold/Eigen/src/Core/Array.h:42
#9 TransInitialProb::Alpha_Beta_Compute_SinglePoint (this=<optimized out>, iOrderNum=<optimized out>, Delta=...) at ./IBDFold/InitalLambda_class.h:310
#10 0x00000000004812fd in MultiplyMethodsComputaionResultOutput(std::string, std::string, std::string, std::string, std::string, int, int, short, int, int, float, int, int, bool const&, bool const&, float, float, int, int, long, double, int, int, std::string&, float, std::string, unsigned long&, int const&) [clone ._omp_fn.0] () at ./IBDFold/TriCoreProgram.cpp:2116
#11 0x00007ffff6fae06a in ?? () from /usr/lib/x86_64-linux-gnu/libgomp.so.1
#12 0x00007ffff6b78b50 in start_thread (arg=<optimized out>) at pthread_create.c:304
#13 0x00007ffff68c2a7d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#14 0x0000000000000000 in ?? ()
run 2:
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff30ac700 (LWP 18465)]
__memmove_ssse3 () at ../sysdeps/x86_64/multiarch/memcpy-ssse3.S:2825
2825 ../sysdeps/x86_64/multiarch/memcpy-ssse3.S: No such file or directory.
(gdb) bt
#0 __memmove_ssse3 () at ../sysdeps/x86_64/multiarch/memcpy-ssse3.S:2825
#1 0x000000000042fa4c in __copy_m<float> (__result=0x7fffb7f58010, __last=<optimized out>, __first=<optimized out>) at /usr/include/c++/4.7/bits/stl_algobase.h:366
#2 __copy_move_a<false, float const*, float*> (__last=<optimized out>, __result=0x7fffb7f58010, __first=<optimized out>) at /usr/include/c++/4.7/bits/stl_algobase.h:384
#3 __copy_move_a2<false, __gnu_cxx::__normal_iterator<float const*, std::vector<float> >, float*> (__result=0x7fffb7f58010, __last=..., __first=...) at /usr/include/c++/4.7/bits/stl_algobase.h:422
#4 copy<__gnu_cxx::__normal_iterator<float const*, std::vector<float> >, float*> (__result=0x7fffb7f58010, __last=..., __first=...) at /usr/include/c++/4.7/bits/stl_algobase.h:454
#5 __uninit_copy<__gnu_cxx::__normal_iterator<float const*, std::vector<float> >, float*> (__result=0x7fffb7f58010, __last=..., __first=...) at /usr/include/c++/4.7/bits/stl_uninitialized.h:95
#6 uninitialized_copy<__gnu_cxx::__normal_iterator<float const*, std::vector<float> >, float*> (__result=0x7fffb7f58010, __last=..., __first=...) at /usr/include/c++/4.7/bits/stl_uninitialized.h:119
#7 __uninitialized_copy_a<__gnu_cxx::__normal_iterator<float const*, std::vector<float> >, float*, float> (__result=0x7fffb7f58010, __last=..., __first=...)
at /usr/include/c++/4.7/bits/stl_uninitialized.h:260
#8 vector (__x=..., this=0x7fffe079c100) at /usr/include/c++/4.7/bits/stl_vector.h:310
#9 _Construct<std::vector<float>, std::vector<float> > (__value=..., __p=0x7fffe079c100) at /usr/include/c++/4.7/bits/stl_construct.h:85
#10 std::__uninitialized_copy<false>::__uninit_copy<std::vector<float, std::allocator<float> >*, std::vector<float, std::allocator<float> >*> (__first=0x7fffc474a480, __last=__last@entry=0x7fffc474b020,
__result=__result@entry=0x7fffe079b4a0) at /usr/include/c++/4.7/bits/stl_uninitialized.h:77
#11 0x000000000042fc20 in uninitialized_copy<std::vector<float>*, std::vector<float>*> (__result=<optimized out>, __last=0x7fffc474b020, __first=<optimized out>)
at /usr/include/c++/4.7/bits/stl_uninitialized.h:119
#12 __uninitialized_copy_a<std::vector<float>*, std::vector<float>*, std::vector<float> > (__result=<optimized out>, __last=0x7fffc474b020, __first=<optimized out>)
at /usr/include/c++/4.7/bits/stl_uninitialized.h:260
#13 __uninitialized_move_if_noexcept_a<std::vector<float>*, std::vector<float>*, std::allocator<std::vector<float> > > (__result=<optimized out>, __last=0x7fffc474b020, __first=<optimized out>,
__alloc=...) at /usr/include/c++/4.7/bits/stl_uninitialized.h:283
#14 std::vector<std::vector<float, std::allocator<float> >, std::allocator<std::vector<float, std::allocator<float> > > >::_M_insert_aux (this=this@entry=0x7fffffffc9c0, __position=..., __x=...)
at /usr/include/c++/4.7/bits/vector.tcc:360
#15 0x00000000004224e3 in push_back (__x=..., this=0x7fffffffc9c0) at /usr/include/c++/4.7/bits/stl_vector.h:893
#16 KinshipIBC_Construct (MultiplyRowNameListKin=..., MultiplyRowNameListIBC=..., MultiplyRowKinship=..., MultiplyIBC=..., SNP_NumEachPairInChrKin=..., SNP_NumEachPairInChrIBC=..., Alpha_BetaProb9=...,
Alpha_BetaProb2=..., iPairList=..., sStudyPairNameList=..., iPair=@0x7ffff30ab890: 258, ibcCheck=@0x7fffffffd570: 0) at ./IBDFold/CommonInformTreat.cpp:845
#17 0x000000000048136a in MultiplyMethodsComputaionResultOutput(std::string, std::string, std::string, std::string, std::string, int, int, short, int, int, float, int, int, bool const&, bool const&, float, float, int, int, long, double, int, int, std::string&, float, std::string, unsigned long&, int const&) [clone ._omp_fn.0] () at ./IBDFold/TriCoreProgram.cpp:2140
#18 0x00007ffff6fae06a in ?? () from /usr/lib/x86_64-linux-gnu/libgomp.so.1
#19 0x00007ffff6b78b50 in start_thread (arg=<optimized out>) at pthread_create.c:304
#20 0x00007ffff68c2a7d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#21 0x0000000000000000 in ?? ()
run 3:
[same bt stack size as initial report, same line had the error condition]
run 4:
Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7ffff10a8700 (LWP 18494)]
0x00007ffff681a475 in *__GI_raise (sig=<optimized out>) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
64 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0 0x00007ffff681a475 in *__GI_raise (sig=<optimized out>) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1 0x00007ffff681d6f0 in *__GI_abort () at abort.c:92
#2 0x00007ffff749a89d in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#3 0x00007ffff7498996 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4 0x00007ffff7497aa9 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5 0x00007ffff74985e1 in __gxx_personality_v0 () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6 0x00007ffff6d9dd5b in ?? () from /lib/x86_64-linux-gnu/libgcc_s.so.1
#7 0x00007ffff6d9e1b8 in _Unwind_Resume () from /lib/x86_64-linux-gnu/libgcc_s.so.1
#8 0x000000000042257c in KinshipIBC_Construct (MultiplyRowNameListKin=..., MultiplyRowNameListIBC=..., MultiplyRowKinship=..., MultiplyIBC=..., SNP_NumEachPairInChrKin=..., SNP_NumEachPairInChrIBC=...,
Alpha_BetaProb9=..., Alpha_BetaProb2=..., iPairList=..., sStudyPairNameList=..., iPair=@0x7ffff10a7890: 258, ibcCheck=@0x7fffffffd570: 0) at ./IBDFold/CommonInformTreat.cpp:846
#9 0x000000000048136a in MultiplyMethodsComputaionResultOutput(std::string, std::string, std::string, std::string, std::string, int, int, short, int, int, float, int, int, bool const&, bool const&, float, float, int, int, long, double, int, int, std::string&, float, std::string, unsigned long&, int const&) [clone ._omp_fn.0] () at ./IBDFold/TriCoreProgram.cpp:2140
#10 0x00007ffff6fae06a in ?? () from /usr/lib/x86_64-linux-gnu/libgomp.so.1
#11 0x00007ffff6b78b50 in start_thread (arg=<optimized out>) at pthread_create.c:304
#12 0x00007ffff68c2a7d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#13 0x0000000000000000 in ?? ()
I changed all the vector lookups in the function to .at() statements, which makes sure they do range checking and produce exceptions when an out-of-array access is attempted. No exeptions were thrown, suggesting that the issue is more likely due to 'MultiplyRowNameListKin'.
Adding in a debug statement, I notice that this happens when the size of the MultiplyRowNameListKin vector is 256. This is somewhat suspicious.
Looking back up the program stack (e.g. #19 on the initial bug report), MultiplyRowNameListKin is constructed on line 1974 of TriCoreProgram.cpp, and only modified once on line 2140 (the problem function). However, MultiplyRowNameListKin is a double vector. It is possible that the outside container has been initialised, but not the inside containers, or something else like that.
This is starting to look like a race condition due to threading. Another debug statement shows that the vector capacity at the time of the crash is 256. It is possible that memory (on my computer) is allocated for up to 256 positions in the vector, and a substantial reallocation is required after this point.
The vector changing *does* happen inside a threaded section, and I notice that the previous '#pragma omp critical' (possibly due to a previous segmentation fault) has been commented out. Removing that comment (i.e. making that line a critical section that can be only performed by one thread at a time) does fix the issue, but it would be nice to do this with threading if possible.
Make KinshipIBC_Construct method thread-critical
I've added a patch for this that implements the fix made in my recent comment, because it looks like this hadn't already been done in v3.003. My attempts to shift the blame further into the KinshipIBC_Construct method were somewhat fruitless -- I surrounded what I thought were all the modification methods with critical blocks, but still got segfaults from the code.
We have improved them.