Re: [wgs-assembler-users] appending to a gkpStore?

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 422-6466

whups forgot to cc. the mail list on my earlier reply

--
Brad Langhorst, Ph.D.
Applications and Product Development Scientist

> On Mar 20, 2015, at 4:41 PM, Langhorst, Brad <Lan...@ne...> wrote:
> 
> HI Brian:
> 
>> Do you have any details on the out of memory error?  It shouldn't be using much memory for fastq files (opposed to the 'fry' format which WILL use gobs of memory to detect duplicate IDs).  The read metadata is kept in core, but just about every process in the assembler will need to do that too.
> 
> The system was loaded when I tried to run the first time:
> Processing INNIE SANGER QV encoding reads from:
>      '/mnt/ngswork/langhorst/deer_assembly/ovi_run15.1.fastq'
>  and '/mnt/ngswork/langhorst/deer_assembly/ovi_run15.2.fastq'
> Could not calloc memory (536870912 * 32 bytes = 17179869184)
> gatekeeper: AS_UTL_alloc.C:49: void* safe_calloc(size_t, size_t): Assertion `p != __null' failed.
> 
>> 1.5 billion reads will be difficult to get assembled with overlaps.  Have you looked into masurca?  This will build 'super reads' from the PE, effectively compressing the data before assembling.
> 
> I’m trying to build unitigs to feed to eautils.
> 
> 80% of these PE reads overlap (based on the result of a stitch with PEAR) but I thought maybe CA would work better with the PE information and all reads.
> 
> I’m running again now that the other jobs have finished, maybe it will complete.
> It looks like I’m enabling duplicate removal in frg file (see below). Didn’t mean to do that. Should I disable that feature by just flipping the 1 to a 0?
> 
> Should I give up on CA and switch to masurca?
> 
> 
> Brad
> 
> Here’s one of the frg files:
> {VER
> ver:2
> }
> {LIB
> act:A
> acc:run15
> ori:I
> mea:280.000
> std:100.000
> src:
> .
> nft:20
> fea:
> forceBOGunitigger=1
> isNotRandom=0
> doNotTrustHomopolymerRuns=0
> doTrim_initialNone=0
> doTrim_initialMerBased=1
> doTrim_initialFlowBased=0
> doTrim_initialQualityBased=0
> doRemoveDuplicateReads=1
> doTrim_finalLargestCovered=1
> doTrim_finalEvidenceBased=0
> doTrim_finalBestEdge=0
> doRemoveSpurReads=1
> doRemoveChimericReads=1
> doCheckForSubReads=0
> doConsensusCorrection=0
> forceShortReadFormat=1
> constantInsertSize=0
> fastqQualityValues=sanger
> fastqOrientation=innie
> fastqMates=/mnt/ngswork/langhorst/deer_assembly/ovi_run15.1.fastq,/mnt/ngswork/langhorst/deer_assembly/ovi_run15.2.fastq
> .
> }
> {VER
> ver:1
> }
> 
>> On Mar 20, 2015, at 4:28 PM, Walenz, Brian <wa...@nb...> wrote:
>> 
>> Unfortunately, appending broke, and we never had a need to fix it.
>> 
>> 
>> 
>> 
>> b
>> 
>> 
>> ________________________________________
>> From: Langhorst, Brad [Lan...@ne...]
>> Sent: Friday, March 20, 2015 1:43 PM
>> To: wgs...@li...
>> Subject: [wgs-assembler-users] appending to a gkpStore?
>> 
>> Hi:
>> 
>> I ran into a problem creating a gkpStore from 3 frg files (pointing to PE fastq files, each about 500M reads)
>> It failed due to a memory allocation error when i tried to import all 3 at once, so I thought I’d try to append to the store like this:
>> $gatekeeper -o $store -T -F $frg_path/run15.frg
>> $gatekeeper -a -o $store -T -F $frg_path/run16.frg
>> $gatekeeper -a -o $store -T -F $frg_path/run17.frg
>> 
>> The first one succeeds, but the append fails immediately.
>> 
>> Seems like the store is somehow marked read-only. I didn’t expect that since the first command succeeded.
>> Should appending to a store work?
>> Should I try an older gatekeeper? Will that cause trouble later if i try to use 8.3 for following steps?
>> 
>> Here’s the log:
>> 
>> Starting file '/mnt/galaxy/data/langhorst/deer_unitigs/run15.frg'.
>> 
>> Processing INNIE SANGER QV encoding reads from:
>>     '/mnt/ngswork/langhorst/deer_assembly/ovi_run15.1.fastq'
>> and '/mnt/ngswork/langhorst/deer_assembly/ovi_run15.2.fastq'
>> 
>> 
>> GKP finished with 1 alerts or errors:
>> 1       # LIB Alert: stddev too big for mean; reset stddev to 0.1 * mean.
>> 
>> 
>> Starting file '/mnt/galaxy/data/langhorst/deer_unitigs/run16.frg'.
>> gatekeeper: AS_PER_genericStore.C:425: int64 appendStringStore(StoreStruct*, char*, uint32): Assertion `s->readOnly == false' failed.
>> 
>> …
>> 
>> [0] /home/NEB/langhorst/wgs-8.3rc1/Linux-amd64/bin/gatekeeper::AS_UTL_catchCrash(int, siginfo*, void*) + 0x2a  [0x42587a]
>> [1] /lib/x86_64-linux-gnu/libpthread.so.0::(null) + 0x10340  [0x7ffb82749340]
>> [2] /lib/x86_64-linux-gnu/libc.so.6::(null) + 0x39  [0x7ffb823aacc9]
>> [3] /lib/x86_64-linux-gnu/libc.so.6::(null) + 0x148  [0x7ffb823ae0d8]
>> [4] /lib/x86_64-linux-gnu/libc.so.6::(null) + 0x2fb86  [0x7ffb823a3b86]
>> [5] /lib/x86_64-linux-gnu/libc.so.6::(null) + 0x2fc32  [0x7ffb823a3c32]
>> [6] /home/NEB/langhorst/wgs-8.3rc1/Linux-amd64/bin/gatekeeper() [0x43129d]
>> [7] /home/NEB/langhorst/wgs-8.3rc1/Linux-amd64/bin/gatekeeper::gkStore::gkStore_addUID(char*) + 0x13f  [0x436d6f]
>> [8] /home/NEB/langhorst/wgs-8.3rc1/Linux-amd64/bin/gatekeeper::AS_UID_load(char*) + 0x196  [0x4254b6]
>> [9] /home/NEB/langhorst/wgs-8.3rc1/Linux-amd64/bin/gatekeeper::GetUID(char*, _IO_FILE*) + 0x11  [0x4264d1]
>> [10] /home/NEB/langhorst/wgs-8.3rc1/Linux-amd64/bin/gatekeeper() [0x42f902]
>> [11] /home/NEB/langhorst/wgs-8.3rc1/Linux-amd64/bin/gatekeeper::ReadProtoMesg_AS(_IO_FILE*, GenericMesg**) + 0x4aa  [0x42719a]
>> [12] /home/NEB/langhorst/wgs-8.3rc1/Linux-amd64/bin/gatekeeper::(null) + 0x5c1  [0x4087b1]
>> [13] /lib/x86_64-linux-gnu/libc.so.6::(null) + 0xf5  [0x7ffb82395ec5]
>> [14] /home/NEB/langhorst/wgs-8.3rc1/Linux-amd64/bin/gatekeeper::(null) + 0xf1  [0x406949]
>> 
>> ------------------------------------------------------------------------------
>> Dive into the World of Parallel Programming The Go Parallel Website, sponsored
>> by Intel and developed in partnership with Slashdot Media, is your hub for all
>> things parallel software development, from weekly thought leadership blogs to
>> news, videos, case studies, tutorials and more. Take a look and join the
>> conversation now. http://goparallel.sourceforge.net/
>> _______________________________________________
>> wgs-assembler-users mailing list
>> wgs...@li...
>> https://lists.sourceforge.net/lists/listinfo/wgs-assembler-users
> 