#3 bug report:Floating point exception

[Andrew]

[xwb@node8 xa1326]$ gdb ../bin/cortex_var_63_c203
GNU gdb Fedora (6.8-37.el5)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu"...
(gdb) run --kmer_size 41 --mem_height 17 --mem_width 100 --remove_seq_errors --colour_list colour_list --dump_binary BGI-203-all-clean.ctx
Starting program: /public3/xwb/solexa_seq/BGI_NCGR/cortex_var_1.0.5.3/bin/cortex_var_63_c203 --kmer_size 41 --mem_height 17 --mem_width 100 --remove_seq_errors --colour_list colour_list --dump_binary BGI-203-all-clean.ctx

-----
Sat Jan 14 05:05:08 2012
Starting Cortex
Command: /public3/xwb/solexa_seq/BGI_NCGR/cortex_var_1.0.5.3/bin/cortex_var_63_c203 --kmer_size 41 --mem_height 17 --mem_width 100 --remove_seq_errors --colour_list colour_list --dump_binary BGI-203-all-clean.ctx
colour list is set to colour_list
Maximum k-mer size (compile-time setting): 63
Actual K-mer size: 41
Hash table created, number of buckets: 131072

-----
Sat Jan 14 05:05:08 2012

-----
Sat Jan 14 05:05:08 2012
List of colours: colour_list (contains one filelist per colour). Load data into consecutive colours starting at 0
Loaded next binary; total kmers in graph is now 22767
Loaded next binary; total kmers in graph is now 32366
Loaded next binary; total kmers in graph is now 37411
Loaded next binary; total kmers in graph is now 41366
Loaded next binary; total kmers in graph is now 48047
Loaded next binary; total kmers in graph is now 52526
Loaded next binary; total kmers in graph is now 55879
Loaded next binary; total kmers in graph is now 59175
Loaded next binary; total kmers in graph is now 64060
Loaded next binary; total kmers in graph is now 66626
Loaded next binary; total kmers in graph is now 69666
Loaded next binary; total kmers in graph is now 75362
Loaded next binary; total kmers in graph is now 77083
Loaded next binary; total kmers in graph is now 81297
Loaded next binary; total kmers in graph is now 83661
Loaded next binary; total kmers in graph is now 86083
Loaded next binary; total kmers in graph is now 89550
Loaded next binary; total kmers in graph is now 91017
Loaded next binary; total kmers in graph is now 93822
Loaded next binary; total kmers in graph is now 96385
Loaded next binary; total kmers in graph is now 99694
Loaded next binary; total kmers in graph is now 101107
Loaded next binary; total kmers in graph is now 102291
Loaded next binary; total kmers in graph is now 105138
Loaded next binary; total kmers in graph is now 106929
Loaded next binary; total kmers in graph is now 108591
Loaded next binary; total kmers in graph is now 110976
Loaded next binary; total kmers in graph is now 113177
Loaded next binary; total kmers in graph is now 115458
Loaded next binary; total kmers in graph is now 117983
Loaded next binary; total kmers in graph is now 122441
Loaded next binary; total kmers in graph is now 124451
Loaded next binary; total kmers in graph is now 125681
Loaded next binary; total kmers in graph is now 126878
Loaded next binary; total kmers in graph is now 129357
Loaded next binary; total kmers in graph is now 131531
Loaded next binary; total kmers in graph is now 133402
Loaded next binary; total kmers in graph is now 136230
Loaded next binary; total kmers in graph is now 139239
Loaded next binary; total kmers in graph is now 141905
Loaded next binary; total kmers in graph is now 143275
Loaded next binary; total kmers in graph is now 146672
Loaded next binary; total kmers in graph is now 149789
Loaded next binary; total kmers in graph is now 152071
Loaded next binary; total kmers in graph is now 155431
Loaded next binary; total kmers in graph is now 157177
Loaded next binary; total kmers in graph is now 158641
Loaded next binary; total kmers in graph is now 161144
Loaded next binary; total kmers in graph is now 162721
Loaded next binary; total kmers in graph is now 165230
Loaded next binary; total kmers in graph is now 167404
Loaded next binary; total kmers in graph is now 171258
Loaded next binary; total kmers in graph is now 174242
Loaded next binary; total kmers in graph is now 176664
Loaded next binary; total kmers in graph is now 178109
Loaded next binary; total kmers in graph is now 179621
Loaded next binary; total kmers in graph is now 183080
Loaded next binary; total kmers in graph is now 184308
Loaded next binary; total kmers in graph is now 186840
Loaded next binary; total kmers in graph is now 189077
Loaded next binary; total kmers in graph is now 190685
Loaded next binary; total kmers in graph is now 193500
Loaded next binary; total kmers in graph is now 195646
Loaded next binary; total kmers in graph is now 199109
Loaded next binary; total kmers in graph is now 200801
Loaded next binary; total kmers in graph is now 203224
Loaded next binary; total kmers in graph is now 204817
Loaded next binary; total kmers in graph is now 206443
Loaded next binary; total kmers in graph is now 207823
Loaded next binary; total kmers in graph is now 210392
Loaded next binary; total kmers in graph is now 211673
Loaded next binary; total kmers in graph is now 213965
Loaded next binary; total kmers in graph is now 216451
Loaded next binary; total kmers in graph is now 219080
Loaded next binary; total kmers in graph is now 221138
Loaded next binary; total kmers in graph is now 222421
Loaded next binary; total kmers in graph is now 223871
Loaded next binary; total kmers in graph is now 226932
Loaded next binary; total kmers in graph is now 228422
Loaded next binary; total kmers in graph is now 229918
Loaded next binary; total kmers in graph is now 232287
Loaded next binary; total kmers in graph is now 234695
Loaded next binary; total kmers in graph is now 236351
Loaded next binary; total kmers in graph is now 237901
Loaded next binary; total kmers in graph is now 239630
Loaded next binary; total kmers in graph is now 242343
Loaded next binary; total kmers in graph is now 245181
Loaded next binary; total kmers in graph is now 246607
Loaded next binary; total kmers in graph is now 250624
Loaded next binary; total kmers in graph is now 253823
Loaded next binary; total kmers in graph is now 256329
Loaded next binary; total kmers in graph is now 257659
Loaded next binary; total kmers in graph is now 262796
Loaded next binary; total kmers in graph is now 264864
Loaded next binary; total kmers in graph is now 267321
Loaded next binary; total kmers in graph is now 269634
Loaded next binary; total kmers in graph is now 273173
Loaded next binary; total kmers in graph is now 277487
Loaded next binary; total kmers in graph is now 278522
Loaded next binary; total kmers in graph is now 281063
Loaded next binary; total kmers in graph is now 285782
Loaded next binary; total kmers in graph is now 287787
Loaded next binary; total kmers in graph is now 290553
Loaded next binary; total kmers in graph is now 293259
Loaded next binary; total kmers in graph is now 295971
Loaded next binary; total kmers in graph is now 297148
Loaded next binary; total kmers in graph is now 298921
Loaded next binary; total kmers in graph is now 300536
Loaded next binary; total kmers in graph is now 301999
Loaded next binary; total kmers in graph is now 304524
Loaded next binary; total kmers in graph is now 305434
Loaded next binary; total kmers in graph is now 306990
Loaded next binary; total kmers in graph is now 308284
Loaded next binary; total kmers in graph is now 310515
Loaded next binary; total kmers in graph is now 311637
Loaded next binary; total kmers in graph is now 313175
Loaded next binary; total kmers in graph is now 314616
Loaded next binary; total kmers in graph is now 316389
Loaded next binary; total kmers in graph is now 317761
Loaded next binary; total kmers in graph is now 319910
Loaded next binary; total kmers in graph is now 322054
Loaded next binary; total kmers in graph is now 323139
Loaded next binary; total kmers in graph is now 324855
Loaded next binary; total kmers in graph is now 327307
Loaded next binary; total kmers in graph is now 328925
Loaded next binary; total kmers in graph is now 331554
Loaded next binary; total kmers in graph is now 332922
Loaded next binary; total kmers in graph is now 335022
Loaded next binary; total kmers in graph is now 337108
Loaded next binary; total kmers in graph is now 338717
Loaded next binary; total kmers in graph is now 339977
Loaded next binary; total kmers in graph is now 341436
Loaded next binary; total kmers in graph is now 343882
Loaded next binary; total kmers in graph is now 345598
Loaded next binary; total kmers in graph is now 346811
Loaded next binary; total kmers in graph is now 348513
Loaded next binary; total kmers in graph is now 350895
Loaded next binary; total kmers in graph is now 352561
Loaded next binary; total kmers in graph is now 354833
Loaded next binary; total kmers in graph is now 356768
Loaded next binary; total kmers in graph is now 358614
Loaded next binary; total kmers in graph is now 359984
Loaded next binary; total kmers in graph is now 362040
Loaded next binary; total kmers in graph is now 362901
Loaded next binary; total kmers in graph is now 365440
Loaded next binary; total kmers in graph is now 366804
Loaded next binary; total kmers in graph is now 368567
Loaded next binary; total kmers in graph is now 370585
Loaded next binary; total kmers in graph is now 371982
Loaded next binary; total kmers in graph is now 373144
Loaded next binary; total kmers in graph is now 374375
Loaded next binary; total kmers in graph is now 375606
Loaded next binary; total kmers in graph is now 377440
Loaded next binary; total kmers in graph is now 378810
Loaded next binary; total kmers in graph is now 379565
Loaded next binary; total kmers in graph is now 380437
Loaded next binary; total kmers in graph is now 381815
Loaded next binary; total kmers in graph is now 382664
Loaded next binary; total kmers in graph is now 384275
Loaded next binary; total kmers in graph is now 385551
Loaded next binary; total kmers in graph is now 386886
Loaded next binary; total kmers in graph is now 388872
Loaded next binary; total kmers in graph is now 390189
Loaded next binary; total kmers in graph is now 390924
Loaded next binary; total kmers in graph is now 392236
Loaded next binary; total kmers in graph is now 393605
Loaded next binary; total kmers in graph is now 395231
Loaded next binary; total kmers in graph is now 396548
Loaded next binary; total kmers in graph is now 397668
Loaded next binary; total kmers in graph is now 399317
Loaded next binary; total kmers in graph is now 400101
Loaded next binary; total kmers in graph is now 401453
Loaded next binary; total kmers in graph is now 402692
Loaded next binary; total kmers in graph is now 404067
Loaded next binary; total kmers in graph is now 405598
Loaded next binary; total kmers in graph is now 407399
Loaded next binary; total kmers in graph is now 409371
Loaded next binary; total kmers in graph is now 412168
Loaded next binary; total kmers in graph is now 413327
Loaded next binary; total kmers in graph is now 414596
Loaded next binary; total kmers in graph is now 415702
Loaded next binary; total kmers in graph is now 417744
Loaded next binary; total kmers in graph is now 419690
Loaded next binary; total kmers in graph is now 421309
Loaded next binary; total kmers in graph is now 422236
Loaded next binary; total kmers in graph is now 423203
Loaded next binary; total kmers in graph is now 424510
Loaded next binary; total kmers in graph is now 425970
Loaded next binary; total kmers in graph is now 427800
Loaded next binary; total kmers in graph is now 429053
Loaded next binary; total kmers in graph is now 430986
Loaded next binary; total kmers in graph is now 431656
Loaded next binary; total kmers in graph is now 433148
Loaded next binary; total kmers in graph is now 434068
Loaded next binary; total kmers in graph is now 435313
Loaded next binary; total kmers in graph is now 436008
Loaded next binary; total kmers in graph is now 437131
Loaded next binary; total kmers in graph is now 438891
Loaded next binary; total kmers in graph is now 439999
Loaded next binary; total kmers in graph is now 441646
Loaded next binary; total kmers in graph is now 443600
Loaded next binary; total kmers in graph is now 445225
Loaded next binary; total kmers in graph is now 446503

-----
Sat Jan 14 05:05:55 2012
Finished loading single_colour binaries
Total kmers in table: 446503
****************************************
SUMMARY:
Colour: MeanReadLen TotalSeq
0 89 78019
1 90 57510
2 89 37844
3 90 46710
4 90 76590
5 90 46800
6 89 50215
7 90 43830
8 89 67769
9 89 42630
10 90 39780
11 90 78120
12 90 35820
13 90 75600
14 90 44370
15 90 38700
16 89 41989
17 90 32850
18 90 46980
19 90 46710
20 89 43367
21 89 36945
22 90 34740
23 90 73170
24 90 37350
25 89 34649
26 89 49409
27 90 45180
28 90 43200
29 90 75420
30 89 74069
31 90 38070
32 89 39484
33 90 34200
34 90 81450
35 89 44899
36 90 45990
37 90 72810
38 90 82980
39 90 74340
40 90 42750
41 90 70560
42 90 61380
43 89 37584
44 90 81360
45 90 39150
46 90 40950
47 90 72450
48 90 47700
49 90 70110
50 90 67770
51 89 46528
52 89 43289
53 89 45179
54 90 38160
55 89 38969
56 89 67127
57 90 47340
58 90 43110
59 90 44280
60 90 42930
61 90 76230
62 90 36630
63 89 69929
64 90 46440
65 90 39690
66 89 43729
67 90 45180
68 90 37080
69 90 54810
70 90 43020
71 90 68130
72 90 42300
73 90 42750
74 90 41130
75 90 44640
76 90 39150
77 90 76140
78 89 38758
79 89 44502
80 90 50040
81 90 49320
82 89 45505
83 90 46710
84 90 40320
85 89 72406
86 89 67112
87 90 38160
88 90 72990
89 90 50850
90 89 77273
91 90 41940
92 89 84025
93 89 42438
94 90 37890
95 89 43556
96 90 66780
97 90 64710
98 90 37440
99 89 49588
100 89 81865
101 89 47471
102 89 74827
103 90 42930
104 89 43369
105 89 37490
106 89 78566
107 89 43559
108 89 55328
109 89 67675
110 89 40308
111 89 46438
112 89 37348
113 89 86787
114 89 43065
115 90 42750
116 89 39973
117 89 66775
118 90 37170
119 89 65887
120 90 48600
121 90 34110
122 89 43092
123 89 74695
124 90 41670
125 90 44280
126 90 42840
127 89 42659
128 90 63360
129 90 42210
130 90 37800
131 89 40072
132 90 66780
133 90 44550
134 90 42390
135 90 44820
136 90 82620
137 90 50040
138 90 81000
139 90 43920
140 90 66060
141 89 45334
142 90 46440
143 90 38520
144 90 70020
145 89 39775
146 90 48240
147 89 75878
148 90 39510
149 89 42479
150 90 36090
151 90 44910
152 89 57868
153 89 41759
154 89 34829
155 89 37168
156 89 47879
157 90 34920
158 90 40950
159 90 42390
160 89 43019
161 89 72088
162 89 51029
163 89 32939
164 90 34200
165 89 44367
166 89 58409
167 89 48059
168 90 40500
169 90 45090
170 90 36450
171 90 47160
172 90 38160
173 90 47250
174 89 41218
175 90 61110
176 89 76565
177 90 77220
178 90 35190
179 90 44010
180 90 45000
181 90 72990
182 90 81090
183 90 65970
184 90 37080
185 90 35280
186 90 41400
187 90 41040
188 89 74937
189 90 43380
190 90 74610
191 90 32670
192 90 41850
193 90 44010
194 90 43560
195 90 36630
196 90 54270
197 90 86400
198 90 48690
199 90 45990
200 89 70998
201 90 51480
202 90 47250
****************************************

-----
Sat Jan 14 05:05:55 2012
Remove nodes that look like sequencing errors. Clip tips first
Thesh is 1

Program received signal SIGFPE, Arithmetic exception.
0x000000000043331c in get_total_depth_of_coverage_across_colours (ginfo=0x7fffffff0b20, genome_length=0) at src/cortex_var/core/graph_info.c:110
110 if (total_seq/genome_length<1)
(gdb) bt
#0 0x000000000043331c in get_total_depth_of_coverage_across_colours (ginfo=0x7fffffff0b20, genome_length=0) at src/cortex_var/core/graph_info.c:110
#1 0x000000000041e7f6 in prune_supernode_if_it_looks_like_is_induced_by_singlebase_errors (node=0x2aaaab117010) at src/cortex_var/core/dB_graph_population.c:3888
#2 0x000000000040e451 in hash_table_traverse_returning_sum (f=0x7ffffffef374, hash_table=0x64d090) at src/hash_table/open_hash/hash_table.c:221
#3 0x000000000041e76e in db_graph_remove_supernodes_more_likely_errors_than_sampling (db_graph=0x64d090, ginfo=0x7fffffff0b20, model_info=0x7fffffff0af0,
max_length_allowed_to_prune=10000, sum_of_covgs_in_desired_colours=0x409814 <element_get_covg_union_of_all_covgs>,
get_edge_of_interest=0x409727 <element_get_colour_union_of_all_colours>, apply_reset_to_specified_edges=0x41cc63 <apply_reset_to_specific_edge_in_union_of_all_colours>,
apply_reset_to_specified_edges_2=0x41cca7 <apply_reset_to_all_edges_in_union_of_all_colours>, max_expected_sup=10000, manual_threshold=1)
at src/cortex_var/core/dB_graph_population.c:3910
#4 0x0000000000407506 in main (argc=12, argv=0x7fffffffe0f8) at src/cortex_var/many_colours/cortex_var.c:805
(gdb) quit

[Zamin Iqbal]

Hi Andrew
OK, this is a clear bug, division by zero, and I apologise. This wil be fixed in the next release, but for now, you can avoid the bug very simply.
In your command-line, remove
--remove_seq_errors
and replace it with
--remove_low_coverage_supernodes N

where N is a threshold of your choice. If you use N=1, then you effectively get the same as remove_seq_errors (but without the bug). You have over 200 samples, each with tiny overage - 50kb or so. How big is your target/genome?
At the end of the supplementary material of our paper is a section describing how you can choose the right parameters based on your experimental design. I am happy to discuss this with you on email, or if you prefer to keep it private, then just play with the threshold N

regards
Zam

[Zamin Iqbal]

Hi Andrew

This is not a bug, but is just a confusion about single and multicolour graphs. I'll explain:

> I have sequencing data of 203 samples. I builded uncleaned individual
>graph for each sample and then merged all of the individual binaries into a
>pooled graph with the parameter "--remove_low_coverage_supernodes 1". I
>want to clean each individual by comparing it with the cleaned pool graph.

If you load fasta, cortex always dumps a single-colour graph, independent of how many colours
you have compiled for.

If you load binaries, then when you use --dump_binary, cortex dumps a graph with the same number of colours as you have compiled for.

So, you correctly built single-colour binaries for your 203 samples.
Then you wanted to pool them, so that you could clean the pool. The right thing to do was to use a ONE-COLOUR graph,
and load them all into one colour, and then clean that single colour graph. Then you could load that 1-colour graph using
Cortex compiled for 2-colours, leaving one colour spare for loading and cleaning each sample one at a time.

Unfortunately, what you did was to use a 201-colour graph and load each sample into a different colour.
You then cleaned that and dumped a 201-colour graph.
Then you tried reloading that 201-colour graph using Cortex compiled for 2 colours

That's why Cortex complained and said
"You are loading binary with 203 colours into a graph with 2 colours - incompatible".

All you need to do is make a single list of all your sample binaries, and load that into a single-colour
COrtex, and clean them again, and dump a 1-colour binary. Then try you final cleaning commandline again

regards

Zam

[Zamin Iqbal]

SORRY. I pasted the last comment in the wrong bug report. This IS a bug. The fix is going to be a bit brutal - I am going to remove --remove_seq_errors in the next release. It's not recommended anyway