Menu
▾
▴
#285 failure to create the overlap store
When running an assembly of ~580 MB of reads, runCA fails. Relevant part of asm.out:
---------------------------------------START Fri Oct 10 11:39:52 2014
/panfs/pstor.storage/rcclocal/zcluster/pacbiotoca/Linux-amd64/bin/overlapStoreBuild -o /home/krdlab/elowry/B2try2/B2asm/asm.ovlStore.BUILDING -g /home/krdlab/elowry/B2try2/B2asm/asm.gkpStore -M 8192 -L /home/krdlab/elowry/B2try2/B2asm/asm.ovlStore.list > /home/krdlab/elowry/B2try2/B2asm/asm.ovlStore.err 2>&1
sh: line 1: 19834 Aborted (core dumped) /panfs/pstor.storage/rcclocal/zcluster/pacbiotoca/Linux-amd64/bin/overlapStoreBuild -o /home/krdlab/elowry/B2try2/B2asm/asm.ovlStore.BUILDING -g /home/krdlab/elowry/B2try2/B2asm/asm.gkpStore -M 8192 -L /home/krdlab/elowry/B2try2/B2asm/asm.ovlStore.list > /home/krdlab/elowry/B2try2/B2asm/asm.ovlStore.err 2>&1
----------------------------------------END Fri Oct 10 11:39:54 2014 (2 seconds)
ERROR: Failed with signal ABRT (6)
================================================================================
runCA failed.
Stack trace:
at /usr/local/pacbiotoca/latest/bin/runCA line 1568, <J> line 1.
main::caFailure("failed to create the overlap store", "/home/krdlab/elowry/B2try2/B2asm/asm.ovlStore.err") called at /usr/local/pacbiotoca/latest/bin/runCA line 3912
main::createOverlapStore() called at /usr/local/pacbiotoca/latest/bin/runCA line 6475
Last few lines of the relevant log file (/home/krdlab/elowry/B2try2/B2asm/asm.ovlStore.err):
/lib64/libc.so.6(abort+0x110)[0x310f631d10]
/lib64/libc.so.6(assert_fail+0xf6)[0x310f629706]
/panfs/pstor.storage/rcclocal/zcluster/pacbiotoca/Linux-amd64/bin/overlapStoreBuild(main+0x2202)[0x407812]
/lib64/libc.so.6(libc_start_main+0xf4)[0x310f61d9c4]
/panfs/pstor.storage/rcclocal/zcluster/pacbiotoca/Linux-amd64/bin/overlapStoreBuild(__gxx_personality_v0+0xf1)[0x405349]
Backtrace (demangled):
[0] /panfs/pstor.storage/rcclocal/zcluster/pacbiotoca/Linux-amd64/bin/overlapStoreBuild::AS_UTL_catchCrash(int, siginfo, void) + 0x26 [0x40add6]
[1] /lib64/libpthread.so.0 [0x311020eca0]
[2] /lib64/libc.so.6::(null) + 0x35 [0x310f630265]
[3] /lib64/libc.so.6::(null) + 0x110 [0x310f631d10]
[4] /lib64/libc.so.6::(null) + 0xf6 [0x310f629706]
[5] /panfs/pstor.storage/rcclocal/zcluster/pacbiotoca/Linux-amd64/bin/overlapStoreBuild::(null) + 0x2202 [0x407812]
[6] /lib64/libc.so.6::(null) + 0xf4 [0x310f61d9c4]
[7] /panfs/pstor.storage/rcclocal/zcluster/pacbiotoca/Linux-amd64/bin/overlapStoreBuild::(null) + 0xf1 [0x405349]
GDB:
Failure message:
failed to create the overlap store
**Plus the overlap_partition.err in 1-overlapper:
HASH: 0 reads or 20000000 length.
REF: 5000000 reads or 0 length.
Loading lengths of 71697 fragments (0mb)
Partitioning for hash: 1-71697 ref: 1,71697
HASH 1- 71697 REFR 1- 71697 STRINGS 71697 BASES 72957 JOB 1
Worked on other data sets so not sure why it isn't working now...thanks!!
Can you post more of file '/home/krdlab/elowry/B2try2/B2asm/asm.ovlStore.err'? This looks like a crash in the threading library, which is odd, because this isn't threaded code.
Changing parameters to the sort might get around the problem. ovlStoreMemory is the only exposed parameter. Halving or doubling is suggested.
Can you recompile with debug support?
Thank you for the quick response!
Here is the asm.oveStore.err:
Scanning overlap files to count the number of overlaps.
Found 10.264 million overlaps.
Memory limit 16046MB supplied. I'll put 13366330 IIDs (841.27 million overlaps) into each of 1 buckets.
bucketizing /home/krdlab/elowry/B2try2/tempB2/./1-overlapper/001/000001.ovb.gz
safeWrite()-- Write failure on AS_OVS_outputOverlap: Disk quota exceeded
safeWrite()-- Wanted to write 262140 objects (size=4), wrote 3344.
overlapStoreBuild: AS_UTL_fileIO.C:67: void AS_UTL_safeWrite(FILE, const void, const char, size_t, size_t): Assertion `(__errno_location ()) == 0' failed.
Failed with 'Aborted'
Backtrace (mangled):
/panfs/pstor.storage/rcclocal/zcluster/pacbiotoca/Linux-amd64/bin/overlapStoreBuild(_Z17AS_UTL_catchCrashiP7siginfoPv+0x26)[0x40add6]
/lib64/libpthread.so.0[0x37ac00eca0]
/lib64/libc.so.6(gsignal+0x35)[0x37ab430265]
/lib64/libc.so.6(abort+0x110)[0x37ab431d10]
/lib64/libc.so.6(assert_fail+0xf6)[0x37ab429706]
/panfs/pstor.storage/rcclocal/zcluster/pacbiotoca/Linux-amd64/bin/overlapStoreBuild(_Z16AS_UTL_safeWriteP8_IO_FILEPKvPKcmm+0x102)[0x40a112]
/panfs/pstor.storage/rcclocal/zcluster/pacbiotoca/Linux-amd64/bin/overlapStoreBuild(_Z19AS_OVS_writeOverlapP17BinaryOverlapFileP10OVSoverlap+0xd2)[0x412752]
/panfs/pstor.storage/rcclocal/zcluster/pacbiotoca/Linux-amd64/bin/overlapStoreBuild(gxx_personality_v0+0x218)[0x405470]
/panfs/pstor.storage/rcclocal/zcluster/pacbiotoca/Linux-amd64/bin/overlapStoreBuild(main+0x905)[0x405f15]
/lib64/libc.so.6(libc_start_main+0xf4)[0x37ab41d9c4]
/panfs/pstor.storage/rcclocal/zcluster/pacbiotoca/Linux-amd64/bin/overlapStoreBuild(gxx_personality_v0+0xf1)[0x405349]
Backtrace (demangled):
[0] /panfs/pstor.storage/rcclocal/zcluster/pacbiotoca/Linux-amd64/bin/overlapStoreBuild::AS_UTL_catchCrash(int, siginfo, void) + 0x26 [0x40add6]
[1] /lib64/libpthread.so.0 [0x37ac00eca0]
[2] /lib64/libc.so.6::(null) + 0x35 [0x37ab430265]
[3] /lib64/libc.so.6::(null) + 0x110 [0x37ab431d10]
[4] /lib64/libc.so.6::(null) + 0xf6 [0x37ab429706]
[5] /panfs/pstor.storage/rcclocal/zcluster/pacbiotoca/Linux-amd64/bin/overlapStoreBuild::AS_UTL_safeWrite(_IO_FILE, void const, char const, unsigned long, unsigned long) + 0x102 [0x40a112]
[6] /panfs/pstor.storage/rcclocal/zcluster/pacbiotoca/Linux-amd64/bin/overlapStoreBuild::AS_OVS_writeOverlap(BinaryOverlapFile, OVSoverlap*) + 0xd2 [0x412752]
[7] /panfs/pstor.storage/rcclocal/zcluster/pacbiotoca/Linux-amd64/bin/overlapStoreBuild::(null) + 0x218 [0x405470]
[8] /panfs/pstor.storage/rcclocal/zcluster/pacbiotoca/Linux-amd64/bin/overlapStoreBuild::(null) + 0x905 [0x405f15]
[9] /lib64/libc.so.6::(null) + 0xf4 [0x37ab41d9c4]
[10] /panfs/pstor.storage/rcclocal/zcluster/pacbiotoca/Linux-amd64/bin/overlapStoreBuild::(null) + 0xf1 [0x405349]
GDB:
Here's your problem:
safeWrite()-- Write failure on AS_OVS_outputOverlap: Disk quota exceeded
We should probably catch that and report a more useful message, instead of looking like a crash.
Ah I see.
I reran the program on a queue with 1/2 TB memory, but it still crashes and now I am getting a different error message unfortunately. Here is the general error message:
runCA failed.
Stack trace:
at /panfs/pstor.storage/rcclocal/zcluster/pacbiotoca/Linux-amd64/bin/runCA line 1568.
main::caFailure("can't set both ovlRefBlockSize and ovlRefBlockLength", undef) called at /panfs/pstor.storage/rcclocal/zcluster/pacbiotoca/Linux-amd64/bin/runCA line 3708
main::createOverlapJobs("normal") called at /panfs/pstor.storage/rcclocal/zcluster/pacbiotoca/Linux-amd64/bin/runCA line 6473
Failure message:
can't set both ovlRefBlockSize and ovlRefBlockLength
----------------------------------------END Wed Oct 15 08:19:20 2014 (0 seconds)
Failed to execute /panfs/pstor.storage/rcclocal/zcluster/pacbiotoca/Linux-amd64/bin/runCA -s /home/krdlab/elowry/B2try2//tempB2/B2.spec -p asm -d . ovlMerThreshold=6552 ovlHashLibrary=2 ovlRefLibrary=1-1 ovlCheckLibrary=1 obtHashLibrary=1-1 obtRefLibrary=1-1 obtCheckLibrary=0 sgePropagateHold="pBcR_asm" stopAfter=overlapper
And here is the content from asm.gkpStore.err:
Starting file '/home/krdlab/elowry/B2try2/B2_pairs.frg'.
Processing INNIE SANGER QV encoding reads from:
'/panfs/pstor.storage/grphomes/krdlab/elowry/B2try2/./B2_pairs.fq' (INTERLACED)
Starting file '/home/krdlab/elowry/B2try2//tempB2/B2.frg'.
Processing SINGLE-ENDED SANGER QV encoding reads from:
'/home/krdlab/elowry/B2try2/Data_for_Sample_B2_018_018127_data_filtered_subreads.fastq'
GKP finished with 1 alerts or errors:
1 # LIB Alert: stddev too big for mean; reset stddev to 0.1 * mean.
I really appreciate your help!
Hopefully you just unset ovlRefBlockLength! The gatekeeper warning can be ignored.
I changed the 'crash' on disk full to a more gentle failure.