Re: [wgs-assembler-users] Configuration SGE

[Ludovic M. <lud...@un...>]

Hi,
Not the best expert, but to me, virtual_free allow the job to swap, 
which you should try to avoid.
and I think h_vmem is the hard limit, so the job would be killed 
whenever the line is crossed.

from http://gridengine.eu/grid-engine-internals
"hard limitation: All processes of the job combined are limited from the 
Linux kernel that they are able to use only the requested amount of 
memory. Further malloc() calls will fail."

whether h_vmem is hard by default if GE has to be checked again, but I'd 
rather use mem_free instead

Best,
ludovic

On 19/01/15 02:22, Miguel Grau wrote:
> Dear all,
>
> I am having some troubles to config wgs 8.2 assembler with SGE options.
> I always get a malloc memory error and I am not sure why. I am working
> with 3 paired fastq files (6 files in total) with 100b length reads (15
> million reads in each fastq file). My config file:
>
> useGrid          = 1
> scriptOnGrid     = 1
>
> sge                   = -A assembly
> sgeMerTrim            = -l h_vmem=150G -l virtual_free=150G
> sgeScript             = -l h_vmem=50G -l virtual_free=50G
> sgeOverlap            =  -l h_vmem=100G -l virtual_free=100G
> sgeMerOverlapSeed     = -l h_vmem=100G -l virtual_free=100G
> sgeMerOverlapExtend   = -l h_vmem=100G -l virtual_free=100G
> sgeConsensus          = -l h_vmem=100G -l virtual_free=100G
> sgeFragmentCorrection = -l h_vmem=100G -l virtual_free=100G
> sgeOverlapCorrection  = -l h_vmem=100G -l virtual_free=100G
>
> overlapper     = ovl #Best for illumina
> unitigger        = bogart  #Best for illumina
>
> #For 50GB...
> ovlHashBits        = 28
> ovlHashBlockLength = 480000000
> #100Gb for overlap
> ovlStoreMemory=102400
>
> ovlThreads     = 2
> ovlRefBlockSize  = 7630000
> frgCorrBatchSize = 1000000
> frgCorrThreads   = 8
>
> The error that I have now is:
>
> ------------------------------------------------------------------------------
> bucketizing /reads/a6/0-overlaptrim-overlap/001/000278.ovb.gz
> bucketizing /reads/a6/0-overlaptrim-overlap/001/000276.ovb.gz
> bucketizing /reads/a6/0-overlaptrim-overlap/001/000275.ovb.gz
> bucketizing /reads/a6/0-overlaptrim-overlap/001/000280.ovb.gz
> bucketizing DONE!
> overlaps skipped:
>         1211882406 OBT - low quality
>                  0 DUP - non-duplicate overlap
>                  0 DUP - different library
>                  0 DUP - dedup not requested
> terminate called after throwing an instance of 'std::bad_alloc'
>     what():  std::bad_alloc
>
> Failed with 'Aborted'
>
> Backtrace (mangled):
>
> /miquel/wgs-8.2/Linux-amd64/bin/overlapStoreBuild(_Z17AS_UTL_catchCrashiP7siginfoPv+0x27)[0x40a697]
> /lib64/libpthread.so.0[0x3ff1c0f710]
> /lib64/libc.so.6(gsignal+0x35)[0x3ff1432925]
> /lib64/libc.so.6(abort+0x175)[0x3ff1434105]
> ....
> ----------------------------------------------------------------------------------
>
> Some idea for the best config?
>
> Thank you,
>
>
> Miguel
>
>
>
>
>
>
> ------------------------------------------------------------------------------
> New Year. New Location. New Benefits. New Data Center in Ashburn, VA.
> GigeNET is offering a free month of service with a new server in Ashburn.
> Choose from 2 high performing configs, both with 100TB of bandwidth.
> Higher redundancy.Lower latency.Increased capacity.Completely compliant.
> http://p.sf.net/sfu/gigenet
> _______________________________________________
> wgs-assembler-users mailing list
> wgs...@li...
> https://lists.sourceforge.net/lists/listinfo/wgs-assembler-users