From: mathog <ma...@ca...> - 2015-01-15 18:00:08
|
On 15-Jan-2015 09:02, Brian Walenz wrote: > The option you're looking for is mbtThreads, with a default of 4. > > Also look into option mbtBatchSize, which sets how many reads to > process > per job. The default is 1 million, and you've already got at least 48 > jobs, so this is probably not an issue. (snip) > So, in summary, I don't know why you're not getting multiple CPUs on > these. You can work around the problem by dropping the batch size to > make > jobs with about 8gb memory (smaller than 512/48), then run 48 jobs in > parallel. So many options, so little time. I don't suppose anybody has put together a script that asks for the relevant system and data information and then emits a SPEC file to run at something approximating optimal speed on the equipment at hand? The input would be something like (no doubt I'm leaving out key information): primary node: RAM=, CPU=, DISK= #fill in the max to use, actual could be more cluster: Y # N if none type=older N=10, RAM=, CPU=, DISK= type=newer N=20, RAM=, CPU=, DISK= queue_system=SGE FRG types: 2 #at least 1 Illumina N=3, totalreads= Sanger N=2, totalreads= As it is now, there are a lot of parameters to fiddle with runCA -options | wc 184 <- !!!! which probably all make perfect sense to people experienced with this software but which are fairly mysterious when first encountered In any case, I did try modifying the -t parameter on 0-mertrim/mertrim.sh while the jobs were running, and the new settings "took" as each new job started. The run times were: -t ~minutes 4 22 16 14 40 12-13 So there isn't much to be gained by pushing that parameter up. > You can increase the number of jobs running at once with > mbtConcurrency. Kind of my point about the script, I overlooked that one. I did use merylThreads, but didn't realize that trim and count used different parameters. Concurrency x Threads, that is simultaneous jobs x cpus/job? There are 7 of the former parameters and 6 of the latter. Presumably if I spent a couple of hours reading all the documentation (which for some reason has been loading really, really slowly from sourceforge) I could make a guess at what would probably work best. The hypothetical script I alluded to would be a lot more convenient! Thanks, David Mathog ma...@ca... Manager, Sequence Analysis Facility, Biology Division, Caltech |