#66 Comet seg fault on MacOS

[William S Noble]

This comes up in the performance test running on MacOS 10.8.5 compiled with i686-apple-darwin11-llvm-gcc-4.2 (GCC) 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.11.00):

../../crux comet --parameter-file crux.param --output-dir comet 051708-worm-ASMS-10.ms2 worm+contaminants.fa
INFO: Beginning comet.
INFO: Writing results to output directory 'comet'.
INFO: CPU: Genomes-MacBook-Pro.local
INFO: Sat Nov 23 14:59:32 PST 2013
INFO: Comet version "2013.02 rev. 0"
INFO:
INFO: Search start: 11/23/2013, 02:59:32 PM
INFO: - Load and process input spectra
INFO: - Allocate memory to store results
INFO: - Number of mass-charge spectra loaded: 14298
INFO: - Search progress: 100%
INFO: - Perform post-search analysis
./run-performance-test.sh: line 40: 37644 Segmentation fault: 11 $CRUX $searchtool $params --output-dir $searchtool $ms2 $proteins

I will recompile with DEBUG and see if I can generate a backtrace. I am running Crux 16224 with Sean's patch for calibrate-scores (attached).

[William S Noble]

I have verified that this segfault does not happen under linux.

I also tried running the debug version of crux within gdb on MacOS, but it runs to completion with no seg fault.

The debug version does seg fault, so once I figure out how to turn on creation of core files in MacOS, I should be able to get a backtrace.

[William S Noble]

Still don't know how to get a core file in MacOS, but I ran the command under valgrind on both macos and linux. In neither case did it dump core, but there are a fair number of small read/write errors (see attached log files).

[William S Noble]

Rgh! The seg fault happens when I have core dumps disabled, but as soon as I turn them on, the program runs smoothly to completion (see below).

../../crux comet --parameter-file crux.param --output-dir comet 051708-worm-ASMS-10.ms2 worm+contaminants.fa
INFO: Beginning comet.
INFO: Writing results to output directory 'comet'.
INFO: CPU: Genomes-MacBook-Pro.local
INFO: Sun Nov 24 10:48:12 PST 2013
INFO: Comet version "2013.02 rev. 0"
INFO:
INFO: Search start: 11/24/2013, 10:48:12 AM
INFO: - Load and process input spectra
INFO: - Allocate memory to store results
INFO: - Number of mass-charge spectra loaded: 14298
INFO: - Search progress: 100%
INFO: - Perform post-search analysis
./run-performance-test.sh: line 40: 418 Segmentation fault: 11 $CRUX $searchtool $params --output-dir $searchtool $ms2 $proteins
bash-3.2$ ulimit -c unlimited
bash-3.2$ rm -r comet; ../../crux comet --parameter-file crux.param --output-dir comet 051708-worm-ASMS-10.ms2 worm+contaminants.fa
INFO: Beginning comet.
INFO: Writing results to output directory 'comet'.
INFO: CPU: Genomes-MacBook-Pro.local
INFO: Sun Nov 24 10:57:45 PST 2013
INFO: Comet version "2013.02 rev. 0"
INFO:
INFO: Search start: 11/24/2013, 10:57:45 AM
INFO: - Load and process input spectra
INFO: - Allocate memory to store results
INFO: - Number of mass-charge spectra loaded: 14298
INFO: - Search progress: 100%
INFO: - Perform post-search analysis
INFO: - Write output
INFO: Search end: 11/24/2013, 10:58:42 AM
INFO:
INFO: Elapsed time: 56.6 s
INFO: Finished crux comet.
INFO: Return Code:0
bash-3.2$

[sjoemac]

Could you try the comet.exe in crux-trunk/src/external/build/src/comet (without debugging symbols) and see if that crashes?

[William S Noble]

It does not. Here is the command line:

../../../../src/external/build/src/comet/comet.exe 051708-worm-ASMS-10.ms2

Parameter file is attached. Output looks like this:

bash-3.2$ ../../../../src/external/build/src/comet/comet.exe 051708-worm-ASMS-10.ms2
Comet version "2013.02 rev. 0"

Search start: 11/24/2013, 03:26:31 PM

• Load and process input spectra
• Allocate memory to store results
• Number of mass-charge spectra loaded: 14298
• Search progress: 100%
• Perform post-search analysis
• Write output
  Search end: 11/24/2013, 03:27:12 PM

[sjoemac]

How about a backtrace of crux comet without debugging symbols compiled in?

[William S Noble]

Here it is:

bash-3.2$ gdb ~/proj/crux/trunk/src/c/crux /cores/core.61686
GNU gdb 6.3.50-20050815 (Apple version gdb-1822) (Sun Aug 5 03:00:42 UTC 2012)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "x86_64-apple-darwin"...Reading symbols for shared libraries ... done

Reading symbols for shared libraries . done
Reading symbols for shared libraries ................................. done

0 0x00007fff827be386 in __semwait_signal ()

(gdb) bt

0 0x00007fff827be386 in __semwait_signal ()

1 0x00007fff8cc5e7c8 in nanosleep ()

2 0x00007fff8cc5e6df in usleep ()

3 0x0000000101e15bba in ThreadPool<postanalysisthreaddata*>::WaitForThreads () at basic_string.h:2153</postanalysisthreaddata*>

4 0x0000000101e15187 in CometPostAnalysis::PostAnalysis () at basic_string.h:2153

5 0x0000000101e27162 in CometSearchManager::DoSearch () at basic_string.h:2153

6 0x0000000101c011e5 in CometApplication::main () at basic_string.tcc:685

7 0x0000000101c1bfef in CruxApplicationList::main () at basic_string.tcc:685

8 0x0000000101b59eb1 in main ()

Seems like the problem is with threading, even though I set num threads to 0.

:(

[Charles E. Grant]

I can only get this to occur intermittently on my Mac running OS X 10.8.5.

void CometPostAnalysis::PostAnalysis(int minNumThreads,
int maxNumThreads)
{
if (!g_staticParams.options.bOutputSqtStream)
logout(" - Perform post-search analysis\n");

// Create the thread pool containing g_staticParams.options.iNumThreads,
// each hanging around and sleeping until asked to do a post analysis.
ThreadPool<postanalysisthreaddata *=""> postAnalysisThreadPool(PostAnalysisThreadProc,
minNumThreads, maxNumThreads);</postanalysisthreaddata>

for (int i=0; i<(int)g_pvQuery.size(); i++)
{
PostAnalysisThreadData *pThreadData = new PostAnalysisThreadData(i);
postAnalysisThreadPool.Launch(pThreadData);
}

// Wait for active post analysis threads to complete processing.
postAnalysisThreadPool.WaitForThreads();
}

[Charles E. Grant]

I tried to reproduce this with the current build of Crux downloaded from Sourceforge. I ran the test case ten times and can no longer generate the segfault. Note that we're no longer compiling with the Apple LLVM compiler, and using the GCC compiler from MacPorts.