Re: [wgs-assembler-users] appending to a gkpStore?

[Walenz, B. <wa...@nb...>]

Unfortunately, appending broke, and we never had a need to fix it.

Do you have any details on the out of memory error?  It shouldn't be using much memory for fastq files (opposed to the 'fry' format which WILL use gobs of memory to detect duplicate IDs).  The read metadata is kept in core, but just about every process in the assembler will need to do that too.

1.5 billion reads will be difficult to get assembled with overlaps.  Have you looked into masurca?  This will build 'super reads' from the PE, effectively compressing the data before assembling.

b


________________________________________
From: Langhorst, Brad [Lan...@ne...]
Sent: Friday, March 20, 2015 1:43 PM
To: wgs...@li...
Subject: [wgs-assembler-users] appending to a gkpStore?

Hi:

I ran into a problem creating a gkpStore from 3 frg files (pointing to PE fastq files, each about 500M reads)
It failed due to a memory allocation error when i tried to import all 3 at once, so I thought I’d try to append to the store like this:
$gatekeeper -o $store -T -F $frg_path/run15.frg
$gatekeeper -a -o $store -T -F $frg_path/run16.frg
$gatekeeper -a -o $store -T -F $frg_path/run17.frg

The first one succeeds, but the append fails immediately.

Seems like the store is somehow marked read-only. I didn’t expect that since the first command succeeded.
Should appending to a store work?
Should I try an older gatekeeper? Will that cause trouble later if i try to use 8.3 for following steps?

Here’s the log:

Starting file '/mnt/galaxy/data/langhorst/deer_unitigs/run15.frg'.

Processing INNIE SANGER QV encoding reads from:
      '/mnt/ngswork/langhorst/deer_assembly/ovi_run15.1.fastq'
  and '/mnt/ngswork/langhorst/deer_assembly/ovi_run15.2.fastq'


GKP finished with 1 alerts or errors:
1       # LIB Alert: stddev too big for mean; reset stddev to 0.1 * mean.


Starting file '/mnt/galaxy/data/langhorst/deer_unitigs/run16.frg'.
gatekeeper: AS_PER_genericStore.C:425: int64 appendStringStore(StoreStruct*, char*, uint32): Assertion `s->readOnly == false' failed.

…

[0] /home/NEB/langhorst/wgs-8.3rc1/Linux-amd64/bin/gatekeeper::AS_UTL_catchCrash(int, siginfo*, void*) + 0x2a  [0x42587a]
[1] /lib/x86_64-linux-gnu/libpthread.so.0::(null) + 0x10340  [0x7ffb82749340]
[2] /lib/x86_64-linux-gnu/libc.so.6::(null) + 0x39  [0x7ffb823aacc9]
[3] /lib/x86_64-linux-gnu/libc.so.6::(null) + 0x148  [0x7ffb823ae0d8]
[4] /lib/x86_64-linux-gnu/libc.so.6::(null) + 0x2fb86  [0x7ffb823a3b86]
[5] /lib/x86_64-linux-gnu/libc.so.6::(null) + 0x2fc32  [0x7ffb823a3c32]
[6] /home/NEB/langhorst/wgs-8.3rc1/Linux-amd64/bin/gatekeeper() [0x43129d]
[7] /home/NEB/langhorst/wgs-8.3rc1/Linux-amd64/bin/gatekeeper::gkStore::gkStore_addUID(char*) + 0x13f  [0x436d6f]
[8] /home/NEB/langhorst/wgs-8.3rc1/Linux-amd64/bin/gatekeeper::AS_UID_load(char*) + 0x196  [0x4254b6]
[9] /home/NEB/langhorst/wgs-8.3rc1/Linux-amd64/bin/gatekeeper::GetUID(char*, _IO_FILE*) + 0x11  [0x4264d1]
[10] /home/NEB/langhorst/wgs-8.3rc1/Linux-amd64/bin/gatekeeper() [0x42f902]
[11] /home/NEB/langhorst/wgs-8.3rc1/Linux-amd64/bin/gatekeeper::ReadProtoMesg_AS(_IO_FILE*, GenericMesg**) + 0x4aa  [0x42719a]
[12] /home/NEB/langhorst/wgs-8.3rc1/Linux-amd64/bin/gatekeeper::(null) + 0x5c1  [0x4087b1]
[13] /lib/x86_64-linux-gnu/libc.so.6::(null) + 0xf5  [0x7ffb82395ec5]
[14] /home/NEB/langhorst/wgs-8.3rc1/Linux-amd64/bin/gatekeeper::(null) + 0xf1  [0x406949]

------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
wgs-assembler-users mailing list
wgs...@li...
https://lists.sourceforge.net/lists/listinfo/wgs-assembler-users