Menu
▾
▴
#305 runCA failed
/NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./1-overlapper/overlap.sh 1512 > /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./1-overlapper/001512.out 2>&1
/NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./1-overlapper/overlap.sh 1513 > /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./1-overlapper/001513.out 2>&1
/NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./1-overlapper/overlap.sh 1514 > /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./1-overlapper/001514.out 2>&1
/NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./1-overlapper/overlap.sh 1515 > /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./1-overlapper/001515.out 2>&1
----------------------------------------END CONCURRENT Mon Apr 27 11:52:28 2015 (1177497 seconds)
----------------------------------------START Mon Apr 27 11:52:40 2015
find -L /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./1-overlapper ( -name *ovb.gz -or -name *ovb ) -print > /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./asm.ovlStore.list
----------------------------------------END Mon Apr 27 11:52:42 2015 (2 seconds)
----------------------------------------START Mon Apr 27 11:52:42 2015
/NABIC/HOME/jsnkim/shinig/PacbioOnly/soft/wgs-8.3rc1/Linux-amd64/bin/overlapStoreBuild -o /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./asm.ovlStore.BUILDING -g /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./asm.gkpStore -M 8192 -L /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./asm.ovlStore.list > /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./asm.ovlStore.err 2>&1
----------------------------------------END Mon Apr 27 12:03:38 2015 (656 seconds)
Warning: failed to load Config_git.pl, something strange about this perl...
ERROR: Failed with signal HUP (1)
================================================================================
runCA failed.
Stack trace:
at /NABIC/HOME/jsnkim/shinig/PacbioOnly/soft/wgs-8.3rc1/Linux-amd64/bin/runCA line 1649, <J> line 1515.
main::caFailure("failed to create the overlap store", "/NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec"...) called at /NABIC/HOME/jsnkim/shinig/PacbioOnly/soft/wgs-8.3rc1/Linux-amd64/bin/runCA line 3993
main::createOverlapStore() called at /NABIC/HOME/jsnkim/shinig/PacbioOnly/soft/wgs-8.3rc1/Linux-amd64/bin/runCA line 6556
Last few lines of the relevant log file (/NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./asm.ovlStore.err):
bucketizing /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./1-overlapper/001/000945.ovb.gz
bucketizing /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./1-overlapper/001/000951.ovb.gz
bucketizing /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./1-overlapper/001/000921.ovb.gz
bucketizing /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./1-overlapper/001/000935.ovb.gz
bucketizing /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./1-overlapper/001/000939.ovb.gz
bucketizing /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./1-overlapper/001/000870.ovb.gz
bucketizing /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./1-overlapper/001/000869.ovb.gz
bucketizing /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./1-overlapper/001/000934.ovb.gz
bucketizing /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./1-overlapper/001/000938.ovb.gz
bucketizing /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./1-overlapper/001/000911.ovb.gz
bucketizing /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./1-overlapper/001/000910.ovb.gz
bucketizing /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./1-overlapper/001/000882.ovb.gz
bucketizing /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./1-overlapper/001/000881.ovb.gz
bucketizing /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./1-overlapper/001/000868.ovb.gz
bucketizing /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./1-overlapper/001/000873.ovb.gz
bucketizing /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./1-overlapper/001/000941.ovb.gz
bucketizing /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./1-overlapper/001/000876.ovb.gz
bucketizing /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./1-overlapper/001/000886.ovb.gz
bucketizing /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./1-overlapper/001/000894.ovb.gz
bucketizing /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./1-overlapper/001/000903.ovb.gz
bucketizing /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./1-overlapper/001/000898.ovb.gz
bucketizing /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./1-overlapper/001/000906.ovb.gz
bucketizing /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./1-overlapper/001/000918.ovb.gz
bucketizing /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./1-overlapper/001/000888.ovb.gz
bucketizing /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./1-overlapper/001/000927.ovb.gz
bucketizing /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./1-overlapper/001/000928.ovb.gz
bucketizing /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./1-overlapper/001/000879.ovb.gz
bucketizing /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./1-overlapper/001/000929.ovb.gz
bucketizing /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./1-overlapper/001/000936.ovb.gz
bucketizing /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./1-overlapper/001/000901.ovb.gz
bucketizing /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./1-overlapper/001/000908.ovb.gz
bucketizing /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./1-overlapper/001/000883.ovb.gz
bucketizing /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./1-overlapper/001/000884.ovb.gz
bucketizing /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./1-overlapper/001/000892.ovb.gz
bucketizing /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./1-overlapper/001/000897.ovb.gz
bucketizing /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./1-overlapper/001/000909.ovb.gz
bucketizing /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./1-overlapper/001/000913.ovb.gz
bucketizing DONE!
overlaps skipped:
0 OBT - low quality
0 DUP - non-duplicate overlap
0 DUP - different library
0 DUP - dedup not requested
reading /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./asm.ovlStore.BUILDING/tmp.sort.000 (0)
sorting /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./asm.ovlStore.BUILDING/tmp.sort.000 (8)
writing /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./asm.ovlStore.BUILDING/tmp.sort.000 (49)
reading /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./asm.ovlStore.BUILDING/tmp.sort.001 (64)
sorting /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./asm.ovlStore.BUILDING/tmp.sort.001 (86)
writing /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./asm.ovlStore.BUILDING/tmp.sort.001 (246)
ERROR: Failed to open output file '/NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./asm.ovlStore.BUILDING/0004': Cannot allocate memory
Failure message:
failed to create the overlap store
----------------------------------------END Mon Apr 27 12:03:40 2015 (1179293 seconds)
Failed to execute /NABIC/HOME/jsnkim/shinig/PacbioOnly/soft/wgs-8.3rc1/Linux-amd64/bin/runCA -s /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100//tempec2_pacbio/ec2_pacbio.spec -p asm -d . ovlMerThreshold=1260 ovlHashLibrary=2 ovlRefLibrary=1-1 ovlCheckLibrary=1 obtHashLibrary=1-1 obtRefLibrary=1-1 obtCheckLibrary=0 gridEnginePropagateHold="pBcR_asm" stopAfter=overlapper
(END)
my spec file is
my spec file is
utgErrorLimit = 4.5
cnsErrorRate = 0.25
cgwErrorRate = 0.25
ovlErrorRate = 0.25
merSize=17
merylMemory = 128000000
merylThreads = 16
ovlStoreMemory = 819200
grid info
useGrid = 0
scriptOnGrid = 0
frgCorrOnGrid = 0
ovlCorrOnGrid = 0
sge = -V -S /bin/sh
sge = -V -A assembly
sgeScript = -pe smp 16
sgeConsensus = -pe smp 1
sgeOverlap = -pe smp 4
sgeFragmentCorrection = -pe smp 2
sgeOverlapCorrection = -pe smp 1
ovlMemory=8GB --hashload 0.7
ovlHashBits = 25
ovlThreads = 4
ovlHashBlockLength = 20000000
for mer overlapper
merCompression = 1
merOverlapperSeedBatchSize = 500000
merOverlapperExtendBatchSize = 250000
frgCorrThreads = 2
frgCorrBatchSize = 100000
ovlCorrBatchSize = 100000
non-Grid settings, if you set useGrid to 0 above these will be used
merylMemory = 128000000
merylThreads = 16
ovlStoreMemory = 819200
ovlConcurrency = 6
cnsConcurrency = 16
merOverlapperThreads = 16
merOverlapperSeedConcurrency = 6
merOverlapperExtendConcurrency = 6
frgCorrConcurrency = 8
ovlCorrConcurrency = 16
cnsConcurrency = 16
It looks like your overlap construction ran out of memory. Based on the spec file, it would be trying to use 82GB of ram to build the store. Does the machine it is running on have 80GB available? There is an known bug when you have much higher Illumina coverage than PacBio coverage, the last library (the PacBio sequences) have a large number of overlaps and the overlap store partitioning becomes uneven:
https://sourceforge.net/p/wgs-assembler/mailman/message/32481695/ https://sourceforge.net/p/wgs-assembler/mailman/message/32481695/
The overlapStoreBuild command has an option called -big which lets you work around this issue. You need to update the overlapStoreBuild command to be:
/NABIC/HOME/jsnkim/shinig/PacbioOnly/soft/wgs-8.3rc1/Linux-amd64/bin/overlapStoreBuild -o /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./asm.ovlStore -g /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./asm.gkpStore -M 8192 -big <ID OF="" FIRST="" PACBIO="" READ=""> -L /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./asm.ovlStore.list > /NABIC/HOME/jsnkim/shinig/shot/20141118_pear_I350_R100/tempec2_pacbio/./asm.ovlStore.err 2>&1
To get the ID of the first PacBio read in your dataset you can run gatekeeper -dumpinfo:
libIID bgnIID endIID active deleted mated totLen clrLen libName
0 1 32550 26374 6176 0 4500974 4500974 GLOBAL
0 0 0 0 0 0 0 0 LegacyUnmatedReads
1 1 25000 25000 0 0 2500000 2500000 illumina
2 25001 32550 1374 6176 0 2000974 2000974 lambdaIll
In this case the ID is 25001. Once you’ve manually built the overlap store, you can re-run your original PBcR command and the pipeline will resume at the next step.
I noticed you are running Illumina-based correction on your dataset. This mode is no longer being updated and is not actively supported. The latest CA (8.3) can assemble using only PacBio from as low as 30X (and has some support for as low as 10X of coverage). If you have enough coverage, I’d suggest trying the PacBio-only method as it is faster than Illumina-based correction and generally gives better results. The low coverage support is documented on the wiki:
http://wgs-assembler.sourceforge.net/wiki/index.php/PBcR#Low_Coverage_Assembly http://wgs-assembler.sourceforge.net/wiki/index.php/PBcR#Low_Coverage_Assembly
Sergey
Related
Bugs: #305