Whole-Genome Shotgun Assembler / Bugs / #306 gatekeeper failed with 'Aborted'

Sergey Koren - 2015-05-18

Hi,

The likeliest reason is an empty input fastq file. Can you check the size of JPN1_PB.longest25.fastq file? Can you also send the full output of the correction pipeline along with the tempJPN1_PB/asm.layout.err file?

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Marta Brozynska - 2015-05-18

Sergey, you're right.. JPN1_PB.longest25.fastq is empty... and it appears between my files as symbolic link...

Please find attached the asm.layout.err file. What do you mean by "full output of the correction pipeline"? What exactly would you like to see? I run correction using the following command:

/home/uqmbrozy/wgs-8.3rc1/Linux-amd64/bin/PBcR -length 500 -partitions 200 -genomeSize 390000000 -t 32 -libraryname JPN1_PB -s pacbio.spec -fastq JPN1-filtered_subreads_mer.fastq

and attached pacbio.spec file.

So I guess that what went wrong is the correction itself..

asm.layout.err

pacbio.spec
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Sergey Koren - 2015-05-18

It looks from the asm.layout.err file that the coverage based on overlaps is very low. This is likely why the output is empty, the sequences got filtered during consensus.

How much input coverage do you have in your PacBio reads? When I say the full output, I mean everything the pipeline prints to stdout/stderr as it runs.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Marta Brozynska - 2015-05-18

The estimated coverage I should have is 38x. Not perfect but from what I've read should work.

I attach some output files that could be useful (I reckon). I don't have the console prints anymore.

Please, if you have any suggestions let me know!

Thanks!

JPN1_PB.correction.err

asm.gkpStore.err

asm.gkpStore.info

asm.initialTrim.summary

asm.ovlStore.err

asm.toerase.err

corrected.log

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Sergey Koren - 2015-05-19

It looks from the gatekeeper and overlap outputs that your fragments were properly loaded and the number of overlaps (1K/per read) is reasonable. However, the later steps indicate the average number of overlaps is only 15/read which is much lower than expected for your number of overlaps and coverage.

Does your temporary directory still have the asm.gkpStore and asm.ovlStore folders? In that case, could you run:
gatekeeper -dumpinfo tempJPN1_PB/asm.gkpStore
overlapStore -p 1 tempJPN1_PB/asm.ovlStore CLR

If not, can you send the asm.toerase.out and JPN1_PB.correction.hist as well as the runPartion.sh and runCorrection.sh files. Also, run
grep -i exception tempJPN1_PB/1-overlapper/*.err
to check for any overlap error messages. Finally, if you're able to share your filtered input fastq, I can try to reproduce your error locally.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Marta Brozynska - 2015-05-19

I can’t understand that behaviour either…

Unfortunately, I don’t have these folders anymore. I attach files you asked for.
The grep command does not give me any results but I attach the 1-overlapper folder as well just in case…

I’m happy to share the input file. You can find it here: http://de.iplantcollaborative.org/dl/d/C45D2638-CB77-4473-8D10-35BEB5E3A81C/JPN1-filtered_subreads_mer.fastq.gz

Hope it works and you’ll be able to download it. If not let me know and we’ll think of another way.
It took around 20 hours of running before the error (on 32CPU and 256G of RAM on a single machine).

Thanks a lot again! I do appreciate your support!!!

1-overlapper.tar.gz

JPN1_PB.correction.hist

asm.toerase.out

runCorrection.sh

runPartition.sh

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Sergey Koren - 2015-05-19

I was able to download the data and launch a run.

From all the output files you sent, everything looks OK. However, from the last step, there are only 1500 sequences with any overlaps despite all previous steps indicating a much larger number of overlaps. I've never seen this before. My guess is something happened to either the asm.ovlStore or asm.gkpStore during the run but I will let you know if I am able to reproduce locally.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Marta Brozynska - 2015-05-19

That's great! Please let me know when you come up with something.

I reckon maybe I set some parameters in the pacbio.spec file too stringent/relaxed and it affected final overlaps... I used the recommended options for diploid large genome but maybe some are not adequate in this case...

Thanks!

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Sergey Koren - 2015-05-29

Hi,

I think I've identified the issue. Your spec file specifies:
merSize=22
which is different from the default and too large to identify overlaps between pacbio sequences. The default is 16. If you remove that line from the spec file I think it should fix your issue. I'm running a test now to confirm. You can also remove the line:
doOverlapBasedTrimming=1
as well, it shouldn't hurt anything but it adds some runtime that isn't needed.

Out of curiosity, how did you make your spec file? I didn't think we had any examples changing the merSize from the default but if the documentation isn't clear about that, I'd like to update it.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Marta Brozynska - 2015-05-31

Hi Sergey,

Thanks a lot. I'll try to re-run everything with default merSize. I'll let you know if this resolves the issue.

I reckon I used the value of 22 because it looks like it's suggested on the PBcR website (http://wgs-assembler.sourceforge.net/wiki/index.php/PBcR) in the section "Important Parameter Considerations" when working with diploid genomes. See "If you are working with a diploid genome and would like to merge haplotypes, you can decrease the overlap stringency to allow the assembler to merge more regions as shown in the above example..." and it follows with an example command with merSize=22.

Of course, it's my mistake as I didn't double check all the parameters...

Anyway, hope it works now! Cheers!

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Sergey Koren - 2015-05-31

I can confirm that my run with the default mer worked and I have over 25X of corrected sequences so your re-run should work as well.

The mer of 22 was meant to be specified for the assembly not the correction but I can see how it is confusing so I updated the page to remove that. Thanks.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Marta Brozynska - 2015-06-01

Great news!

However, I run into another problem. As my resources has shrunk a little bit from the last time, using the same spec.file (with default merSize) I got the following:

ERROR: Overlap prep job /ebi/bscratch/uqmbrozy//tempJPN1_PB/1-overlapper/correct_reads_part 1 FAILED. ERROR: Overlap prep job /ebi/bscratch/uqmbrozy//tempJPN1_PB/1-overlapper/correct_reads_part 2 FAILED. ERROR: Overlap prep job /ebi/bscratch/uqmbrozy//tempJPN1_PB/1-overlapper/correct_reads_part 3 FAILED.

and so on…

In 1.hash.err (and the remaining as well) I have the following:

Error occurred during initialization of VM Could not reserve enough space for 73400320KB object heap Command exited with non-zero status 1 0.00user 0.00system 0:00.01elapsed 22%CPU (0avgtext+0avgdata 5532maxresident)k 0inputs+8outputs (0major+1500minor)pagefaults 0swaps

I have 1,928,732 PacBio reads (14779464519 bp), 8 CPUs and 70Gb of RAM. I found out than you can modify ovlHashBits, ovlHashBlockLength and ovlRefBlockSize to resolve hash problems. I tried various combination but can’t get it going.

Please, if you have how I can get rid of this problem, I’d be very grateful. I feel I’m so close to get my assembly running but still having these small errors…
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Sergey Koren - 2015-06-01

It looks like the JVM was trying to reserve 70GB of ram but your machine didn't actually have enough space to reserve that much memory. Is it possible other tasks are running on the machine and using part of the memory. You can update ovlMemory to make the program use less RAM (ovlMemory=48 for example would mean use 48GB). You can also update ovlStoreMemory= 48000
merylMemory = 48000

THe ovlHashBits/ovlHashBLockLength/etc are only for assembly (not correction) and we've moved away from using them too much (the defaults should work reasonably well).

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Marta Brozynska - 2015-06-01

I've limited the memory and it's looking good so far!!!

Thanks a million!!!

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Sergey Koren - 2015-06-08

My assembly and correction finished last week so I can post both the corrected reads and the assembly on my public FTP site if you would like (both reads and asm is 3GB compressed).

The assembly stats are:
Total contigs: 2585
BasesInFasta: 384759810
Min: 9523
Max: 1692155
N50: 219409

This genome is quite repetitive so I think this assembly is OK given the relatively lower coverage.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Marta Brozynska - 2015-06-08

Hi Sergey, wow I didn't know you were still running this assembly... I'm running it too at the moment but love to have the one you did as well. Please upload it to your FTP if it's not a problem.
I knew from some previous analysis that the genome was fairly repetitive so I'm quite happy with the stats :)
Thanks a lot!!!

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Sergey Koren - 2015-06-08

I left it running and forgot about it and checked it this week and saw it was finished. You can download the tgz file here:
https://gembox.cbcb.umd.edu/shared/JPN1.asm.tgz

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Marta Brozynska - 2015-08-25

Hi Sergey. I know it’s been a while since my issues here and I’m sure you don’t remember this… but after your help and your assembly of my species of interest I’ve been trying to reproduce the assembly… however, unsuccessfully. The weird thing I’ve been experiencing is I’m getting a very large assembly in comparison to the one you sent me – almost double the size – and I don’t know how to explain that. I was changing some parameters, adjusting kmer size (from 14 to 22) and other i.e. error rates and error limits. I consulted other resources and other examples as well. I’ve run several assemblies and still I’m getting the size of almost 600Mb whereas it should be around 400Mb – like the one you got (384Mb). Also, I’m using the corrected reads I got from you.

Please, I’d like to know is there is anything that you do differently while running assemblies… Please, find attached the latest spec.file I used (I got the genome size of 660Mb!). Is there anything you’d set up in a different manner? Please, I really would appreciate your advice A LOT! And trust me – you’re me last resource right now :(

pacbio.spec.large

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
- Sergey Koren - 2015-08-27
  
  Hi,
  
  I think this is related to the assembler generating extra contigs for very repetitive genomes. Basically, it generates more than one contig representing the same repeat copy. I believe I filtered your assembly to eliminate these which might explain the size difference between my result and your result. I thought I had documented this in my original reply to you but I’m sorry if I did not.
  
  In many cases, these contigs are primarily one long sequence with a lot of smaller contained sequences, making it look like it has high coverage/high read count but really it’s one anchoring read. We have a utility script to detect this pattern. If you have your assembly in a folder named asm, running:
  <path to="" ca="">/markRepeatUnique -g asm/asm.gkpStore -t asm/asm.tigStore/ 2 -i -1 -span 0.5 -lowcov 2 0.5 -o asm -j 1 -n
  This will mark unitigs with one read encompassing over 50% of the length or having <= 2X coverage for 50% of the length as repetitive noise. It will print total bases left in the unfiltered sequences. You can run the command without options for detailed explanations but they’re rather assembly-developer centric. The output is a report like:
  Command Line options:
  singleReadMaxCoverage 0.500000
  lowCoverage 2 coverage 0.500000 fraction
  minReads 2
  tooLong 4294967295
  tooShort 1000
  Loading fragment data.
  Generating statistics.
  Analyzing statistics.
  Processing unitigs.</path>
  
  classification number of unitigs total length
  unique: 2585 384759810
  singleton: 39846 326946432
  repeat: 11618 184292492
  too few reads: 0 0
  low cov stat: 0 0
  too short: 0 0
  microhet: 0 0
  spanning read: 11618 184292492
  low coverage: 0 0
  
  The number in unique total length should be the true genome size once the noise is removed. Then, you can get a list of matching unitigs and dump their sequences:
  tigStore -d properties -U -t asm.tigStore 2 -g asm.gkpStore |grep -B 5 -a unitigSuggestRepeat |awk '{if ($1 == "maID") { CURR_ID=$NF; } else if ($1 =="unitigSuggestRepeat" && match($2, "0")) { print CURR_ID; }}' > unique.utg
  
  unique.utg.fasta
  for id in cat unique.utg; do
  echo "Dumping $id"
  tigStore -g asm.gkpStore -t asm.tigStore 2 -d consensus -u $id >> unique.utg.fasta
  done
  
  Again, sorry for not documenting this more clearly.
  Serge
  
  On Aug 25, 2015, at 5:51 PM, Marta Brozynska martabrozynska@users.sf.net wrote:
  
  Hi Sergey. I know it’s been a while since my issues here and I’m sure you don’t remember this… but after your help and your assembly of my species of interest I’ve been trying to reproduce the assembly… however, unsuccessfully. The weird thing I’ve been experiencing is I’m getting a very large assembly in comparison to the one you sent me – almost double the size – and I don’t know how to explain that. I was changing some parameters, adjusting kmer size (from 14 to 22) and other i.e. error rates and error limits. I consulted other resources and other examples as well. I’ve run several assemblies and still I’m getting the size of almost 600Mb whereas it should be around 400Mb – like the one you got (384Mb). Also, I’m using the corrected reads I got from you.
  
  Please, I’d like to know is there is anything that you do differently while running assemblies… Please, find attached the latest spec.file I used (I got the genome size of 660Mb!). Is there anything you’d set up in a different manner? Please, I really would appreciate your advice A LOT! And trust me – you’re me last resource right now :(
  
  Attachments:
  
  pacbio.spec.large https://sourceforge.net/p/wgs-assembler/bugs/_discuss/thread/d85ae1b7/a24d/attachment/pacbio.spec.large (612 Bytes; application/octet-stream)
  [bugs:#306] http://sourceforge.net/p/wgs-assembler/bugs/306/ gatekeeper failed with 'Aborted'
  
  Status: open
  Group: gatekeeper
  Created: Mon May 18, 2015 09:57 PM UTC by Marta Brozynska
  Last Updated: Mon Jun 08, 2015 09:40 PM UTC
  Owner: nobody
  Attachments:
  
  asm.gkpStore.err https://sourceforge.net/p/wgs-assembler/bugs/306/attachment/asm.gkpStore.err (2.5 kB; application/octet-stream)
  Hi!
  
  I’m running PBcR (correction + assembly) on PacBio only data on 400Mb genome. I got the following error: (please find attached also the asm.gkpStore.err file mentioned in the error)
  
  /home/uqmbrozy/wgs-8.3rc1/Linux-amd64/bin/runCA -s /ebi/bscratch/uqmbrozy//tempJPN1_PB/JPN1_PB.spec -p asm -d JPN1_PB ovlRefBlockLength=100000000000 ovlRefBlockSize=0 useGrid=0 scriptOnGrid=0 unitigger=bogart ovlErrorRate=0.10 utgErrorRate=0.10 cgwErrorRate=0.10 cnsErrorRate=0.10 utgGraphErrorLimit=3.25 utgGraphErrorRate=0.05 utgMergeErrorLimit=5.25 utgMergeErrorRate=0.05 frgCorrBatchSize=100000 doOverlapBasedTrimming=1 obtErrorRate=0.08 obtErrorLimit=4.5 frgMinLen=0.5 ovlMinLen=40 "batOptions=-RS -NS -CS" consensus=pbutgcns merSize=22 cnsMaxCoverage=1 cnsReuseUnitigs=1 gridEnginePropagateHold="pBcR_asm" JPN1_PB.longest25.frg
  ----------------------------------------START Tue May 19 04:24:38 2015
  /home/uqmbrozy/wgs-8.3rc1/Linux-amd64/bin/gatekeeper -o /ebi/bscratch/uqmbrozy/JPN1_PB/asm.gkpStore.BUILDING -T -F /ebi/bscratch/uqmbrozy/JPN1_PB.longest25.frg > /ebi/bscratch/uqmbrozy/JPN1_PB/asm.gkpStore.err 2>&1
  sh: line 1: 40819 Aborted /home/uqmbrozy/wgs-8.3rc1/Linux-amd64/bin/gatekeeper -o /ebi/bscratch/uqmbrozy/JPN1_PB/asm.gkpStore.BUILDING -T -F /ebi/bscratch/uqmbrozy/JPN1_PB.longest25.frg > /ebi/bscratch/uqmbrozy/JPN1_PB/asm.gkpStore.err 2>&1
  ----------------------------------------END Tue May 19 04:24:39 2015 (1 seconds)
  ERROR: Failed with signal ABRT (6)
  ================================================================================
  
  runCA failed.
  
  Stack trace:
  
  at /home/uqmbrozy/wgs-8.3rc1/Linux-amd64/bin/runCA line 1649
  main::caFailure('gatekeeper failed', '/ebi/bscratch/uqmbrozy/JPN1_PB/asm.gkpStore.err') called at /home/uqmbrozy/wgs-8.3rc1/Linux-amd64/bin/runCA line 1978
  main::preoverlap('/ebi/bscratch/uqmbrozy/JPN1_PB.longest25.frg') called at /home/uqmbrozy/wgs-8.3rc1/Linux-amd64/bin/runCA line 6551
  
  Last few lines of the relevant log file (/ebi/bscratch/uqmbrozy/JPN1_PB/asm.gkpStore.err):
  
  Starting file '/ebi/bscratch/uqmbrozy/JPN1_PB.longest25.frg'.
  
  Processing SINGLE-ENDED SANGER QV encoding reads from:
  '/ebi/bscratch/uqmbrozy//JPN1_PB.fastq'
  
  gatekeeper: AS_PER_gkStore_IID.C:168: void gkStore::gkStore_computeRanges(AS_IID, AS_IID, int64&, int64&, int64&, int64&, int64&, int64&, int64&, int64&, int64&): Assertion `bgnIID <= endIID' failed.
  
  Failed with 'Aborted'
  
  Backtrace (mangled):
  
  /home/uqmbrozy/wgs-8.3rc1/Linux-amd64/bin/gatekeeper(Z17AS_UTL_catchCrashiP7siginfoPv+0x27)[0x426347]
  /lib64/libpthread.so.0(+0xf810)[0x2aaaab681810]
  /lib64/libc.so.6(gsignal+0x35)[0x2aaaab8c1c65]
  /lib64/libc.so.6(abort+0x181)[0x2aaaab8c3241]
  /lib64/libc.so.6(__assert_fail+0xf0)[0x2aaaab8bab20]
  /home/uqmbrozy/wgs-8.3rc1/Linux-amd64/bin/gatekeeper(_ZN7gkStore21gkStore_computeRangesEjjRlS0_S0_S0_S0_S0_S0_S0_S0+0x96b)[0x43fb9b]
  /home/uqmbrozy/wgs-8.3rc1/Linux-amd64/bin/gatekeeper(_ZN8gkStream5resetEjj+0x1a3)[0x43c013]
  /home/uqmbrozy/wgs-8.3rc1/Linux-amd64/bin/gatekeeper(_ZN12gkStoreStats4initEP7gkStore+0xcf)[0x43b72f]
  /home/uqmbrozy/wgs-8.3rc1/Linux-amd64/bin/gatekeeper(_Z22AS_GKP_summarizeErrorsPc+0x4a)[0x41bf6a]
  /home/uqmbrozy/wgs-8.3rc1/Linux-amd64/bin/gatekeeper(main+0x16a3)[0x409e23]
  /lib64/libc.so.6(__libc_start_main+0xe6)[0x2aaaab8adc36]
  /home/uqmbrozy/wgs-8.3rc1/Linux-amd64/bin/gatekeeper[0x407219]
  
  Backtrace (demangled):
  
  [0] /home/uqmbrozy/wgs-8.3rc1/Linux-amd64/bin/gatekeeper::AS_UTL_catchCrash(int, siginfo, void) + 0x27 [0x426347]
  [1] /lib64/libpthread.so.0::(null) + 0xf810 [0x2aaaab681810]
  [2] /lib64/libc.so.6::(null) + 0x35 [0x2aaaab8c1c65]
  [3] /lib64/libc.so.6::(null) + 0x181 [0x2aaaab8c3241]
  [4] /lib64/libc.so.6::(null) + 0xf0 [0x2aaaab8bab20]
  [5] /home/uqmbrozy/wgs-8.3rc1/Linux-amd64/bin/gatekeeper::gkStore::gkStore_computeRanges(unsigned int, unsigned int, long&, long&, long&, long&, long&, long&, long&, long&, long&) + 0x96b [0x43fb9b]
  [6] /home/uqmbrozy/wgs-8.3rc1/Linux-amd64/bin/gatekeeper::gkStream::reset(unsigned int, unsigned int) + 0x1a3 [0x43c013]
  [7] /home/uqmbrozy/wgs-8.3rc1/Linux-amd64/bin/gatekeeper::gkStoreStats::init(gkStore) + 0xcf [0x43b72f]
  [8] /home/uqmbrozy/wgs-8.3rc1/Linux-amd64/bin/gatekeeper::AS_GKP_summarizeErrors(char) + 0x4a [0x41bf6a]
  [9] /home/uqmbrozy/wgs-8.3rc1/Linux-amd64/bin/gatekeeper::(null) + 0x16a3 [0x409e23]
  [10] /lib64/libc.so.6::(null) + 0xe6 [0x2aaaab8adc36]
  [11] /home/uqmbrozy/wgs-8.3rc1/Linux-amd64/bin/gatekeeper() [0x407219]
  
  GDB:
  
  Failure message:
  
  gatekeeper failed
  
  ----------------------------------------END Tue May 19 04:24:39 2015 (1 seconds)
  Failed to execute /home/uqmbrozy/wgs-8.3rc1/Linux-amd64/bin/runCA -s /ebi/bscratch/uqmbrozy//tempJPN1_PB/JPN1_PB.spec -p asm -d JPN1_PB ovlRefBlockLength=100000000000 ovlRefBlockSize=0 useGrid=0 scriptOnGrid=0 unitigger=bogart ovlErrorRate=0.10 utgErrorRate=0.10 cgwErrorRate=0.10 cnsErrorRate=0.10 utgGraphErrorLimit=3.25 utgGraphErrorRate=0.05 utgMergeErrorLimit=5.25 utgMergeErrorRate=0.05 frgCorrBatchSize=100000 doOverlapBasedTrimming=1 obtErrorRate=0.08 obtErrorLimit=4.5 frgMinLen=0.5 ovlMinLen=40 "batOptions=-RS -NS -CS" consensus=pbutgcns merSize=22 cnsMaxCoverage=1 cnsReuseUnitigs=1 gridEnginePropagateHold="pBcR_asm" JPN1_PB.longest25.frg
  uqmbrozy@b11a15:/ebi/bscratch/uqmbrozy>
  
  I’ve been trying to find the solution but can’t find anything relevant to this error. I’d appreciate a piece of advice very much! Let me know if you need more information about the run.
  
  Thanks!
  
  Sent from sourceforge.net because you indicated interest in https://sourceforge.net/p/wgs-assembler/bugs/306/ https://sourceforge.net/p/wgs-assembler/bugs/306/
  To unsubscribe from further messages, please visit https://sourceforge.net/auth/subscriptions/ https://sourceforge.net/auth/subscriptions/
  
  Related
  
  Bugs: #306
  
  If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

gatekeeper failed with 'Aborted'

Group

Searches

Help

#306 gatekeeper failed with 'Aborted'

Related

Discussion

Related