Re: [wgs-assembler-users] CGW having problems with BOGART unitigs

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 422-6466

Hi-

[this ended up in my spam folder, sorry]

I don't see any problems here, but can't explain what it is doing in the
pauses.  There are some steps in scaffold merging that are not interesting
and don't log anything - the fill gaps case was probably this.

In scaffold merging, after an iteration of merging it done, it needs to slog
through the scaffolds and find more work to do.  There isn't much logging
here.  It is possible that the last case is from this.

There is one problem in cgw worth mentioning here.  It can get stuck in an
infinite loop where it merges two scaffolds, decides the new scaffold is
flawed, breaks the new scaffold into two scaffolds, and repeats.  If this
happens, the logs will show repeated merges between scaffolds number A and
A+1.  The sizes of these two scaffolds is constant between iterations.

Is the machine otherwise healthy - is the CGW process size below physical
memory size?

b

On 4/29/14 7:28 AM, "kuhl" <ku...@mo...> wrote:

> Dear CA developers,
>
> I am currently running an assembly of a mammalian genome (2.1Gbp) using
> illumina paired ends and mate pairs and 454 reads. I recently switched
> from
> using BOG to BOGART, which doubled the N50 of the unitigs. Thus, I was
> quite
> euphoric about the results, but now there is a problem in cgw. At some
> scaffolds the assembler runs for hours and hours without any output to the
> logfile. Is this a known problem?
>
> Find below three examples from the cgw logfile.
>
> Regards, Heiner
>
> example 1(during Fill_Gaps iteration):
>
> CreateAContigInScaffold()-- new contig 9629698 in scaffold 4689
> CreateAContigInScaffold()-- new contig 9629699 in scaffold 4689
> WARNING:  Negative variance (379508.35 +- -46149.23) from edgeOrient I CI
> 9571434 F (1179469.65 +- 17520656.72 -- 799558.65 +- 17510779.03) to MI
> 68466 R (1140067.00 +- 17463615.80 -- 1179067.00 +- 17464629.80)NING:
> Negative variance (379508.35 +- -46149.23) from edgeOrient I CI 9571434 F
> (1179469.65 +- 17520656.72 -- 799558.65 +- 17510779.03) to MI 68466 R
> (1140067.00 +- 17463615.80 -- 1179067.00 +- 17464629.80)NING:  Negative
> variance (379508.35 +- -46149.23) from edgeOrient I CI 9571434 F
> (1179469.65 +- 17520656.72 -- 799558.65 +- 17510779.03) to MI 68466 R
> (1140067.00 +- 17463615.80 -- 1179067.00 +- 17464629.80)NING:  Negative
> variance (379508.35 +- -46149.23) from edgeOrient I CI 9571434 F
> (1179469.65 +- 17520656.72 -- 799558.65 +- 17510779.03) to MI 68466 R
> (1140067.00 +- 17463615.80 -- 1179067.00 +- 17464629.80)trf failed with
> info=11; no solution found, giving up.
> RecomputeOffsetsInScaffold() returned RECOMPUTE_LAPACK on scaffold 4701;
> no more work will help.
>
> Here it took several hours until cgw proceeded
>
> MergeMultiAlignsFast_new()-- failed to find overlap between contigs
> 9612721 and 9571434.
> CreateAContigInScaffold()-- MergeMultiAlignsFast_new() failed.
> CreateAContigInScaffold()-- contig 9612721 (original 9612721)
> 847501-614216
> CreateAContigInScaffold()-- contig 9571434 (original 9571434)
> 1179469-799558
> CreateAContigInScaffold()-- contig 9612720 (original 9612720)
> 1047392-847481
> CreateAContigInScaffold()-- contig 9612719 (original 9612719)
> 1054076-1047372
> CreateAContigInScaffold()-- contig 9578740 (original 9578740)
> 1059537-1054056
> CreateAContigInScaffold()-- contig 9612718 (original 9612718)
> 1084352-1059517
> CreateAContigInScaffold()-- contig 9591744 (original 9591744)
> 1103685-1084332
> CreateAContigInScaffold()-- contig 9578739 (original 9578739)
> 1132833-1103665
> CreateAContigInScaffold()-- contig 212091 (original 212091)
> 1132813-1140087
> CreateAContigInScaffold()-- contig 68466 (original 68466)  1140067-1179067
>
> * CIScaffold 4701 numCI:19 (a:75856 b:68466)  length: 1179469
>> Includes CIs
>              0: CI    75856 sid     4701 len    1256,     32.7 ends
> 0,   1256 var       0.0,     32.7 orient F
>              1: CI   169716 sid     4701 len   11486,    298.6 ends
> 1236,  12722 var 11663292.9,11663591.6 orient F
>              2: CI   244409 sid     4701 len    2610,     67.9 ends
> 12702,  15312 var 12466586.4,12466654.3 orient F
>              3: CI    78245 sid     4701 len   21682,    563.7 ends
> 36974,  15292 var 14686531.7,14685968.0 orient R
>              4: CI  9578742 sid     4701 len  239019,   6214.5 ends
> 277524,  38505 var 15137021.4,15130806.9 orient R
>              5: CI  9612724 sid     4701 len   76477,   1988.4 ends
> 354054, 277577 var 15143369.0,15141380.6 orient R
>              6: CI  9612723 sid     4701 len   96573,   2510.9 ends
> 450889, 354316 var 15148845.4,15146334.6 orient R
>              7: CI  9612722 sid     4701 len  129181,   3358.7 ends
> 580050, 450869 var 15166458.9,15163100.2 orient R
>              8: CI  9591745 sid     4701 len   31856,    828.3 ends
> 611886, 580030 var 15826952.4,15826124.1 orient R
>              9: CI  9612721 sid     4701 len  233285,   6065.4 ends
> 847501, 614216 var 16123024.1,16116958.7 orient R
>             10: CI  9571434 sid     4701 len  379911,   9877.7 ends
> 1179469, 799558 var 17520656.7,17510779.0 orient R
>             11: CI  9612720 sid     4701 len  199911,   5197.7 ends
> 1047392, 847481 var 16161345.7,16156148.0 orient R
>             12: CI  9612719 sid     4701 len    6704,    174.3 ends
> 1054076,1047372 var 16696287.7,16696113.4 orient R
>             13: CI  9578740 sid     4701 len    5481,    142.5 ends
> 1059537,1054056 var 16931871.5,16931729.0 orient R
>             14: CI  9612718 sid     4701 len   24835,    645.7 ends
> 1084352,1059517 var 17365259.4,17364613.7 orient R
>             15: CI  9591744 sid     4701 len   19353,    503.2 ends
> 1103685,1084332 var 17370622.7,17370119.5 orient R
>             16: CI  9578739 sid     4701 len   29168,    758.4 ends
> 1132833,1103665 var 17391605.6,17390847.2 orient R
>             17: CI   212091 sid     4701 len    7274,    189.1 ends
> 1132813,1140087 var 17414697.6,17414886.7 orient F
> ______________________________________________________________________________
> ________________________________________
>
> example 2(during Fill_Gaps iteration):
>
> * CIScaffold 4670 numCI:2 (a:75476 b:9629697)  length: 3296
>> Includes CIs
>              0: CI    75476 sid     4670 len    1480,     38.5 ends
> 1480,      0 var      38.5,      0.0 orient R
>              1: CI  9629697 sid     4670 len    2692,     70.0 ends
> 604,   3296 var    9114.7,   9184.7 orient F
>>  M  Edges A
>           cidA:4670 cidB:5344 weight:2    ori:O con:0 distance:-1668
> stddev:150.706  (144551240,144551241)
>           cidA:4670 cidB:7729 weight:2    ori:A con:0 distance:43
> stddev:134.686  (119529581,119529580)
>           cidA:4670 cidB:8383 weight:1    ori:O con:0 distance:-177
> stddev:190.49  (143054256,143054255)
>           cidA:4670 cidB:8383 weight:7    ori:O con:1 distance:-1387
> stddev:80.549  (148871104,148871105)
>           cidA:4670 cidB:14282 weight:2    ori:A con:0 distance:-184
> stddev:134.697  (159606342,159606343)
>           cidA:4670 cidB:39401 weight:1    ori:O con:0 distance:-1032
> stddev:190.549  (10056130,10056129)
>           cidA:4670 cidB:47012 weight:1    ori:A con:0 distance:-270
> stddev:190.497  (163142964,163142963)
>           cidA:4670 cidB:47012 weight:1    ori:A con:0 distance:-1831
> stddev:213.14  (40610662,40610663)
>           cidA:4670 cidB:47012 weight:4    ori:A con:0 distance:-971
> stddev:100.436  (150398128,150398127)
>           cidA:4670 cidB:47525 weight:2    ori:A con:0 distance:-470
> stddev:134.711  (64096137,64096138)
>           cidA:4670 cidB:48709 weight:3    ori:A con:0 distance:-1828
> stddev:109.954  (3540936,3540935)
>>  M  Edges B
>           cidA:4670 cidB:30766 weight:1    ori:N con:0 distance:279
> stddev:830.551  (200817167,200817166)
>           cidA:4670 cidB:40711 weight:5    ori:N con:0 distance:-754
> stddev:85.2075  (68316430,68316429)
>           cidA:4670 cidB:43356 weight:2    ori:N con:0 distance:-3106
> stddev:150.768  (152889486,152889487)
>
> <---- Here cgw stays for several hours without any message and but using
> 100% CPU
>
> I stopped Fill gaps and restarted cgw at merge scaffolds aggressive
> ________________________________________________________________________
>
> example 3(during merge scaffolds aggressive):
>
> isQualityScaffoldMergingEdge()-- Merge scaffolds 39672 (191435.2bp) and
> 39878 (67088.0bp): gap -3026.4bp +- 96.4bp weight 143 BA_AB edge
> isQualityScaffoldMergingEdge()--   scaffold 39672 instrumenter happy
> 12692.0 gap 3.5 misorient close 0.0 correct 1.0 far 0.0 oriented close
> 130.0 far 11.0 missing 816.8 external 209.8
> isQualityScaffoldMergingEdge()--   scaffold 39878 instrumenter happy
> 4214.0 gap 0.0 misorient close 0.0 correct 1.0 far 0.0 oriented close 48.0
> far 1.0 missing 291.1 external 263.8
> isQualityScaffoldMergingEdge()--   scaffold (new) instrumenter happy
> 16969.0 gap 35.8 misorient close 0.0 correct 2.0 far 0.0 oriented close
> 178.0 far 109.0 missing 1000.0 external 229.2
> isQualityScaffoldMergingEdge()--   before: 0.914 satisfied (16905/1299
> good/bad mates)  after: 0.918 satisfied (16968/1288 good/bad mates)
> isQualityScaffoldMergingEdge()--   ARE happy enough to merge 100 (0.918 >=
> 0.985) || (0.918 >= 0.914) || ((16968 > 16905) && (-0.175 <= 0.300))
> isQualityScaffoldMergingEdge()--  NEW pass (1800/0)
> ExamineSEdgeForUsability_Interleaved()-- Expected end contigs to overlap,
> didn't find it, will not merge.
> InsertScaffoldContentsIntoScaffold()-- Insert scaffold 21 (527789bp) into
> scaffold 49855 (0bp) at offset 0.000 +/- 0.000 orient F
> InsertScaffoldContentsIntoScaffold()-- Insert CI      487  19429bp fwd
>    0 +- 0               19429 +- 505         was         0 +- 0
>   19429 +- 505
> InsertScaffoldContentsIntoScaffold()-- Insert CI  9600072  74365bp fwd
> 19409 +- 11175           93774 +- 13109       was     19409 +- 11175
>
>   93774 +- 13109
> InsertScaffoldContentsIntoScaffold()-- Insert CI  9571242 133049bp fwd
> 93754 +- 46754          226803 +- 50213       was     93754 +- 46754
>
>  226803 +- 50213
> InsertScaffoldContentsIntoScaffold()-- Insert CI  9571243 150819bp fwd
> 227078 +- 209488         377897 +- 213409      was    227078 +- 209488
>
>   377897 +- 213409
> InsertScaffoldContentsIntoScaffold()-- Insert CI  9571244 149912bp fwd
> 377877 +- 228539         527789 +- 232437      was    377877 +- 228539
>
>   527789 +- 232437
> InsertScaffoldContentsIntoScaffold()-- Insert scaffold 43749 (445501bp)
> into scaffold 49855 (527789bp) at offset 527768.962 +/- 489.705 orient R
> InsertScaffoldContentsIntoScaffold()-- Insert CI  9599993  88959bp rev
> 527769 +- 239811         616728 +- 242124      was    445501 +- 569549
>
>   356542 +- 567236
> InsertScaffoldContentsIntoScaffold()-- Insert CI  9599992  14970bp rev
> 616708 +- 249455         631678 +- 249845      was    356562 +- 559904
>
>   341592 +- 559515
> InsertScaffoldContentsIntoScaffold()-- Insert CI  9599991  69788bp rev
> 635107 +- 764942         704895 +- 766757      was    338163 +- 44417
>
>   268375 +- 42603
> InsertScaffoldContentsIntoScaffold()-- Insert CI  9469786    476bp fwd
> 704875 +- 796682         705351 +- 796694      was    268395 +- 12678
>
>   267919 +- 12666
> InsertScaffoldContentsIntoScaffold()-- Insert CI  9599990 267939bp rev
> 705331 +- 802393         973270 +- 809360      was    267939 +- 6966
>
>        0 +- 0
> InsertScaffoldContentsIntoScaffold()-- Insert scaffold 28 (34647bp) into
> scaffold 49856 (0bp) at offset 0.000 +/- 0.000 orient R
> InsertScaffoldContentsIntoScaffold()-- Insert CI      635  34647bp rev
>    0 +- 0               34647 +- 901         was     34647 +- 901
>       0 +- 0
> InsertScaffoldContentsIntoScaffold()-- Insert scaffold 47219 (43488bp)
> into scaffold 49856 (34647bp) at offset 33447.000 +/- 29.489 orient R
> InsertScaffoldContentsIntoScaffold()-- Insert CI  9623655  43488bp rev
> 33447 +- 870             76935 +- 2000        was     43488 +- 1131
>
>       0 +- 0
> CreateAContigInScaffold()-- new contig 9631137 in scaffold 49856
> InsertScaffoldContentsIntoScaffold()-- Insert scaffold 183 (306704bp) into
> scaffold 49857 (0bp) at offset 0.000 +/- 0.000 orient R
> InsertScaffoldContentsIntoScaffold()-- Insert CI  9600994 101398bp rev
>    0 +- 0              101398 +- 2636        was    306704 +- 102182
>  205306 +- 99545
> InsertScaffoldContentsIntoScaffold()-- Insert CI  9600993 107087bp rev
> 101378 +- 14121          208465 +- 16905       was    205326 +- 88061
>
>    98239 +- 85277
> InsertScaffoldContentsIntoScaffold()-- Insert CI  9600992  79255bp rev
> 208445 +- 91912          287700 +- 93973       was     98259 +- 10270
>
>    19004 +- 8209
> InsertScaffoldContentsIntoScaffold()-- Insert CI     3847  19024bp fwd
> 287680 +- 101687         306704 +- 102182      was     19024 +- 495
>
>        0 +- 0
> SetCIScaffoldTLength()-- adjusted scaffold 49857 from (306704 +- 102182)
> to (306704 +- 102182)
> InsertScaffoldContentsIntoScaffold()-- Insert scaffold 39383 (214584bp)
> into scaffold 49857 (306704bp) at offset 306684.000 +/- 320.690 orient F
> InsertScaffoldContentsIntoScaffold()-- Insert CI  9588022 214584bp fwd
> 306684 +- 102842         521268 +- 108421      was         0 +- 0
>
>   214584 +- 5579
> InsertScaffoldContentsIntoScaffold()-- Insert CI  9579474 180065bp fwd
> 340832 +- 104702         520897 +- 109384      was     34148 +- 1860
>
>   214213 +- 6541
> --> here again waiting for at least 2 hours
>
> ---------------------------------------------------------------
> Dr. Heiner Kuhl
> MPI Molecular Genetics            Tel:   + 49 + 30 / 8413 1776
> Next Generation Sequencing
> Ihnestrasse 73                    email: ku...@mo...
> D-14195 Berlin                    http://www.molgen.mpg.de/SeqCore
> ---------------------------------------------------------------
>
>
> ------------------------------------------------------------------------------
> "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
> Instantly run your Selenium tests across 300+ browser/OS combos.  Get
> unparalleled scalability from the best Selenium testing platform available.
> Simple to use. Nothing to install. Get started now for free."
> http://p.sf.net/sfu/SauceLabs
> _______________________________________________
> wgs-assembler-users mailing list
> wgs...@li...
> https://lists.sourceforge.net/lists/listinfo/wgs-assembler-users