Re: [wgs-assembler-users] Problem with Hybrid Assembly using both Illumina and 454 reads

[Walenz, B. <bw...@jc...>]

hi-

It is these two lines:

ovlThreads          = 2
ovlConcurrency                = 24

The first says that each process will use 2 threads, and the second says to run 24 processes at the same time, for a total of 48 cores used.  Dropping ovlConcurrency to 16 should work.

b
--
Brian Walenz
Sr. Software Engineer
J. Craig Venter Institute


On 7/31/12 9:18 AM, "Quan, Xueping" <x....@im...> wrote:

Hi

I am working on a large plant genome (genome size about 3.5Gb), I got about 131Gb Illumina paired-end and 1.8Gb 454 mate pair reads. The assembling is running on a HPC with shared memory with upper memory limit and number of CPUs I could use being 800Gb and 32 cores.
However, the assembling job was killed by the system because "ncpus 33.30 exceeded limit 32" in the  overlapInCore stage. Below is my spec file, could you please have a look to see where it is wrong and how to optimize:

"
#
#  Expected rate of sequencing error. Allow pairwise alignments up to this rate.
#  Sanger reads can use values less than one. Titanium reads require 3% in unitig.
#
utgErrorRate=0.03
utgErrorLimit=2.5  # Allow mismatches over and above the utgErrorRate. This helps with Illumina reads.
ovlErrorRate=0.06 # Larger than utg to allow for correction.
cnsErrorRate=0.10 # Larger than utg to avoid occasional consensus failures
cgwErrorRate=0.10 # Larger than utg to allow contig merges across high-error ends
#
merSize = 22 # default=22; use lower to combine across heterozygosity, higher to separate near-identical repeat copies
overlapper=ovl # the mer overlapper for 454-like data is insensitive to homopolymer problems but requires more RAM and disk
#
unitigger = bog
utgBubblePopping = 1
# utgGenomeSize = 3.5gb
#
#  MERYL calculates K-mer seeds
merylMemory   = 512000
merylThreads    = 24
#
#  OVERLAPPER calculates overlaps
ovlHashBits=25
ovlHashBlockLength=180000000
ovlThreads          = 2
ovlConcurrency                = 24
ovlRefBlockSize  = 32000000
#
#  OVERLAP STORE build the database
#ovlStoreMemory   = 8GB  # Oops! That doesn't work. See correction below.
ovlStoreMemory = 8192 # Mbp
#
# ERROR CORRECTION applied to overlaps
frgCorrThreads    = 10
frgCorrConcurrency = 3
ovlCorrBatchSize  = 1000000
ovlCorrConcurrency = 25
#
# UNITIGGER configuration
#
# CONSENSUS configuration
cnsConcurrency   = 16
"

Thanks very much!

Xueping

Dr. Xueping Quan
Research Associate in BioInformatics
Imperial College London
Tel: +44(0)207 594 17 80
email:x....@im...
Personal:http://www3.imperial.ac.uk/people/x.quan
Group:  www3.imperial.ac.uk/savolainenlab <https://exchange.imperial.ac.uk/ecp/Customize/www3.imperial.ac.uk/savolainenlab>