#234 Could not malloc memory error (PacBioToCA from SMRT 1.3.3)

overlapper
closed-out-of-date
Crash (103)
9
2014-12-02
2013-02-13
No

I am using PacBioToCA shipped as part of SMRT Analysis 1.3.3 and get the following error when run on an 8 core machine with 23 GB reserved memory.
In terms of my data, 70 million paired-end 100 bp Illumina reads are being used to correct 150,000 PacBio reads.

The 000001.out first overlap job looks the following:

STRING_NUM_BITS 31
OFFSET_BITS 31
STRING_NUM_MASK 2147483647
OFFSET_MASK 2147483647
MAX_STRING_NUM 2147483647

Hash_Mask_Bits 24
Max_Hash_Strings 304827
Max_Hash_Data_Len 30000078
Max_Hash_Load 0.750000
Kmer Length 15
Min Overlap Length 40
MAX_ERRORS 8192
ERRORS_FOR_FREE 1

HASH_TABLE_SIZE 16777216
sizeof(Hash_Bucket_t) 216
hash table size: 3456 MB

check 64 MB
info 1 MB
start 2 MB

safeRead()-- Read failure on openStore: Cannot allocate memory.
safeRead()-- Wanted to read 1 objects (size=80), read 1.
overlapInCore: AS_UTL_fileIO.C:112: size_t AS_UTL_safeRead(FILE*, void*, const char*, size_t, size_t): Assertion `(*__errno_location ()) == 0' failed.
/n/scratch03/cavanaugh_lab/odmytrenko/illumina_3/temppacbio/0-overlaptrim-overlap/overlap.sh: line 61: 5295 Aborted (core dumped) $bin/overlapInCore -G --hashbits 24 --hashload 0.75 -t 2 $opt -k 15 -k /n/scratch03/cavanaugh_la$

Then the program runs another overlap 157 jobs. The content of the last output file looks like this:

STRING_NUM_BITS 31
OFFSET_BITS 31
STRING_NUM_MASK 2147483647
OFFSET_MASK 2147483647
MAX_STRING_NUM 2147483647

Hash_Mask_Bits 24
Max_Hash_Strings 12129
Max_Hash_Data_Len 23153461
Max_Hash_Load 0.750000
Kmer Length 15
Min Overlap Length 40
MAX_ERRORS 8192
ERRORS_FOR_FREE 1

HASH_TABLE_SIZE 16777216
sizeof(Hash_Bucket_t) 216
hash table size: 3456 MB

Could not malloc memory (3623878656 bytes)
overlapInCore: AS_UTL_alloc.C:73: void* safe_malloc(size_t): Assertion `p != __null' failed.
/n/scratch03/cavanaugh_lab/odmytrenko/illumina_3/temppacbio/0-overlaptrim-overlap/overlap.sh: line 61: 8097 Aborted (core dumped) $bin/overlapInCore -G --hashbits 24 --hashload 0.75 -t 2 $opt -k 15 -k /n/scratch03/cavanaugh_la$

I am using the following .spec settings.

# original asm settings
utgErrorRate = 0.25
utgErrorLimit = 4.5

cnsErrorRate = 0.25
cgwErrorRate = 0.25
ovlErrorRate = 0.25

merSize = 15

merylMemory = 23000
merylThreads = 8

ovlStoreMemory = 8180

# grid info
useGrid = 0
scriptOnGrid = 0
frgCorrOnGrid = 0
ovlCorrOnGrid = 0

ovlHashBits = 24
ovlThreads = 2
ovlHashBlockLength = 30000000
ovlRefBlockSize = 50000000

frgCorrThreads = 2
frgCorrBatchSize = 100000

ovlCorrBatchSize = 100000

ovlConcurrency = 6
cnsConcurrency = 8
frgCorrConcurrency = 8
ovlCorrConcurrency = 8
cnsConcurrency = 8

I tried changing the parameters, but get the same error. Is it really a bug or the problem resides in the way I choose the parameters in the spec file?
Thank you!

Oleg

Discussion

  • oleg_dmytrenko

    oleg_dmytrenko - 2013-02-13
    • assigned_to: nobody --> jasonmiller9704
    • priority: 5 --> 9
     
  • Sergey Koren

    Sergey Koren - 2013-06-12

    The parameters are overloading the memory of your system with overlap jobs. The parameters above will use approximately 6GB of ram for each overlap process. The machine will try to run 6 of them at a time, thus using 36GB of RAM. With only 23GB of ram available, ovlConcurrency should be set to at most 3 (23/6). To get full use of the 8 CPUs, I recommend setting ovlConcurrency=2 and ovlThreads=4. There is more information on selecting the overlap parameters on the runCA wiki page:
    https://sourceforge.net/apps/mediawiki/wgs-assembler/index.php?title=RunCA#OVL_Overlapper

    One final recommendation would also be to not use any more than 50-100X of Illumina data for correction. Using more will slow down the correction without significantly improving your corrected sequences.

     
  • Sergey Koren

    Sergey Koren - 2014-12-02

    Closing due to inactivity.

     
  • Sergey Koren

    Sergey Koren - 2014-12-02
    • status: open --> closed-out-of-date
     

Log in to post a comment.

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:

JavaScript is required for this form.





No, thanks