Re: [wgs-assembler-users] Configuration SGE

[Brian W. <th...@gm...>]

I've never seen large overlap jobs perform better than small jobs.  Target
an 8gb job with ~4 CPUs each.  My default configuration is:

ovlHashBits         = 22
ovlHashBlockLength  = 200000000
ovlRefBlockSize     = 18000000
ovlThreads          = 6

The two 'hash' sizes control how big the job is.  The 'ref block size'
controls how many reads are processed by each job, i.e., how long the job
runs.

b


On Mon, Jan 19, 2015 at 5:10 AM, Ludovic Mallet <
lud...@un...> wrote:

> Hi,
> Not the best expert, but to me, virtual_free allow the job to swap,
> which you should try to avoid.
> and I think h_vmem is the hard limit, so the job would be killed
> whenever the line is crossed.
>
> from http://gridengine.eu/grid-engine-internals
> "hard limitation: All processes of the job combined are limited from the
> Linux kernel that they are able to use only the requested amount of
> memory. Further malloc() calls will fail."
>
> whether h_vmem is hard by default if GE has to be checked again, but I'd
> rather use mem_free instead
>
> Best,
> ludovic
>
> On 19/01/15 02:22, Miguel Grau wrote:
> > Dear all,
> >
> > I am having some troubles to config wgs 8.2 assembler with SGE options.
> > I always get a malloc memory error and I am not sure why. I am working
> > with 3 paired fastq files (6 files in total) with 100b length reads (15
> > million reads in each fastq file). My config file:
> >
> > useGrid          = 1
> > scriptOnGrid     = 1
> >
> > sge                   = -A assembly
> > sgeMerTrim            = -l h_vmem=150G -l virtual_free=150G
> > sgeScript             = -l h_vmem=50G -l virtual_free=50G
> > sgeOverlap            =  -l h_vmem=100G -l virtual_free=100G
> > sgeMerOverlapSeed     = -l h_vmem=100G -l virtual_free=100G
> > sgeMerOverlapExtend   = -l h_vmem=100G -l virtual_free=100G
> > sgeConsensus          = -l h_vmem=100G -l virtual_free=100G
> > sgeFragmentCorrection = -l h_vmem=100G -l virtual_free=100G
> > sgeOverlapCorrection  = -l h_vmem=100G -l virtual_free=100G
> >
> > overlapper     = ovl #Best for illumina
> > unitigger        = bogart  #Best for illumina
> >
> > #For 50GB...
> > ovlHashBits        = 28
> > ovlHashBlockLength = 480000000
> > #100Gb for overlap
> > ovlStoreMemory=102400
> >
> > ovlThreads     = 2
> > ovlRefBlockSize  = 7630000
> > frgCorrBatchSize = 1000000
> > frgCorrThreads   = 8
> >
> > The error that I have now is:
> >
> >
> ------------------------------------------------------------------------------
> > bucketizing /reads/a6/0-overlaptrim-overlap/001/000278.ovb.gz
> > bucketizing /reads/a6/0-overlaptrim-overlap/001/000276.ovb.gz
> > bucketizing /reads/a6/0-overlaptrim-overlap/001/000275.ovb.gz
> > bucketizing /reads/a6/0-overlaptrim-overlap/001/000280.ovb.gz
> > bucketizing DONE!
> > overlaps skipped:
> >         1211882406 OBT - low quality
> >                  0 DUP - non-duplicate overlap
> >                  0 DUP - different library
> >                  0 DUP - dedup not requested
> > terminate called after throwing an instance of 'std::bad_alloc'
> >     what():  std::bad_alloc
> >
> > Failed with 'Aborted'
> >
> > Backtrace (mangled):
> >
> >
> /miquel/wgs-8.2/Linux-amd64/bin/overlapStoreBuild(_Z17AS_UTL_catchCrashiP7siginfoPv+0x27)[0x40a697]
> > /lib64/libpthread.so.0[0x3ff1c0f710]
> > /lib64/libc.so.6(gsignal+0x35)[0x3ff1432925]
> > /lib64/libc.so.6(abort+0x175)[0x3ff1434105]
> > ....
> >
> ----------------------------------------------------------------------------------
> >
> > Some idea for the best config?
> >
> > Thank you,
> >
> >
> > Miguel
> >
> >
> >
> >
> >
> >
> >
> ------------------------------------------------------------------------------
> > New Year. New Location. New Benefits. New Data Center in Ashburn, VA.
> > GigeNET is offering a free month of service with a new server in Ashburn.
> > Choose from 2 high performing configs, both with 100TB of bandwidth.
> > Higher redundancy.Lower latency.Increased capacity.Completely compliant.
> > http://p.sf.net/sfu/gigenet
> > _______________________________________________
> > wgs-assembler-users mailing list
> > wgs...@li...
> > https://lists.sourceforge.net/lists/listinfo/wgs-assembler-users
>
>
>
> ------------------------------------------------------------------------------
> New Year. New Location. New Benefits. New Data Center in Ashburn, VA.
> GigeNET is offering a free month of service with a new server in Ashburn.
> Choose from 2 high performing configs, both with 100TB of bandwidth.
> Higher redundancy.Lower latency.Increased capacity.Completely compliant.
> http://p.sf.net/sfu/gigenet
> _______________________________________________
> wgs-assembler-users mailing list
> wgs...@li...
> https://lists.sourceforge.net/lists/listinfo/wgs-assembler-users
>