From: mathog <ma...@ca...> - 2015-01-14 17:33:14
|
It looks like I set the wgs spec file parameters wrong, because both mer and mertrim run single threaded (most of the time) on a machine with 48 cpus and 530G of RAM. The data is 4 sets of illumina data. The data was loaded into gatekeeper OK, and that was single threaded too, but I can see why that might be. However, after that it began doing these very, very slowly: /home/wgs_project/do_illumina_wgs/./0-mertrim/mertrim.sh 48 \ > /home/wgs_project/do_illumina_wgs/./0-mertrim/..0048.err 2>&1 Most of the times when I have checked this either mer or mertrim is running at 99% cpu. On some occasions mertrim has gone a bit higher: 14097 wgsuser 39 19 18.7g 16g 1304 S 401.7 3.3 49:43.93 merTrim That is still poor use of this machine, leaving it mostly idle. Here is the spec file, minus all comments: utgErrorRate=0.03 utgErrorLimit=2.5 ovlErrorRate=0.06 cnsErrorRate=0.10 cgwErrorRate=0.10 merSize = 22 overlapper=ovl unitigger = bog utgBubblePopping = 1 merylMemory = 128000 merylThreads = 25 ovlHashBits=25 ovlHashBlockLength=180000000 ovlThreads = 2 ovlConcurrency = 20 ovlRefBlockSize = 32000000 ovlStoreMemory = 8192 # Mbp frgCorrThreads = 10 frgCorrConcurrency = 3 ovlCorrBatchSize = 1000000 ovlCorrConcurrency = 25 cnsConcurrency = 16 useGrid = 0 scriptOnGrid = 0 s_300_qseq.frg s_1000_qseq.frg s_3000_qseq.frg s_5000_qseq.frg What needs to be tweaked to get mer, mertrim to use more of the machine's resources? This spec file was based on a couple found here and there on the web, and I'm sure that many of the memory/threads parameters are not optimal. Thank you, David Mathog ma...@ca... Manager, Sequence Analysis Facility, Biology Division, Caltech |