[wgs-assembler-users] mer, mertrim running single threaded on large SMP machine

[mathog <ma...@ca...>]

It looks like I set the wgs spec file parameters wrong, because both mer 
and mertrim run single threaded (most of the time) on a machine with 48 
cpus and 530G of RAM. The data is 4 sets of illumina data.  The data was 
loaded into gatekeeper OK, and that was single threaded too, but I can 
see why that might be.  However, after that it began doing these very, 
very slowly:

/home/wgs_project/do_illumina_wgs/./0-mertrim/mertrim.sh 48 \
   > /home/wgs_project/do_illumina_wgs/./0-mertrim/..0048.err 2>&1

Most of the times when I have checked this either mer or mertrim is 
running at 99% cpu.  On some occasions mertrim has gone a bit higher:

14097 wgsuser    39  19 18.7g  16g 1304 S 401.7  3.3  49:43.93 merTrim

That is still poor use of this machine, leaving it mostly idle.

Here is the spec file, minus all comments:

utgErrorRate=0.03
utgErrorLimit=2.5
ovlErrorRate=0.06
cnsErrorRate=0.10
cgwErrorRate=0.10
merSize = 22
overlapper=ovl
unitigger = bog
utgBubblePopping = 1
merylMemory   = 128000
merylThreads    = 25
ovlHashBits=25
ovlHashBlockLength=180000000
ovlThreads          = 2
ovlConcurrency                = 20
ovlRefBlockSize  = 32000000
ovlStoreMemory = 8192 # Mbp
frgCorrThreads    = 10
frgCorrConcurrency = 3
ovlCorrBatchSize  = 1000000
ovlCorrConcurrency = 25
cnsConcurrency   = 16
useGrid          = 0
scriptOnGrid     = 0
s_300_qseq.frg
s_1000_qseq.frg
s_3000_qseq.frg
s_5000_qseq.frg

What needs to be tweaked to get mer, mertrim to use more of the 
machine's resources?  This spec file was based on a couple found here 
and there on the web, and I'm sure that many of the memory/threads 
parameters are not optimal.

Thank you,

David Mathog
ma...@ca...
Manager, Sequence Analysis Facility, Biology Division, Caltech