#17 Error: More than 2^31 skim hits would be taken

[Anna Z.]

Hi,

I tried to run a denovo metagenome assembly of an archaeal enrichment
culture with MIRA 4.0.2. I have a total of ~80 million Solexa paired end
reads and ~2.5 million IonTorrent reads.
I used a machine (GNU/Linux) with 48 cores and 0.5 TB RAM, for the
assembly I used 40 cores. The assembly was running nearly 5 days until an
error occurred with a controlled program stop.

This is the manifest I used:

project = mira_assembly_140415
job = genome,denovo,accurate
parameters = -GE:not=40
parameters = -NW:cac=warn # due to high coverage (~300x)
parameters = -HS:ldn=yes # due to high coverage (~300x)

readgroup = IlluminaPairedEnd
autopairing
rename_prefix = HWI-ST1253F_0129 HWI
data = /scratch/anna/Na_*_sorted_prinseq_good.fastq
technology = solexa

readgroup = IonTorrent
data = /scratch/anna/R_201*_goodqual-cutadapt.fastq
technology = iontor

last part of the log file:

========================== Memory self assessment ==============================
Running in 64 bit mode.

Dump from /proc/meminfo

MemTotal: 529423348 kB
MemFree: 610268 kB
Buffers: 3940 kB
Cached: 382586796 kB
SwapCached: 124888 kB
Active: 308976712 kB
Inactive: 213340940 kB
Active(anon): 134129936 kB
Inactive(anon): 5597196 kB
Active(file): 174846776 kB
Inactive(file): 207743744 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 33559548 kB
SwapFree: 32944632 kB
Dirty: 4 kB
Writeback: 0 kB
AnonPages: 139620152 kB
Mapped: 8976 kB
Shmem: 116 kB
Slab: 1153232 kB
SReclaimable: 1087244 kB
SUnreclaim: 65988 kB
KernelStack: 3832 kB
PageTables: 278136 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 298271220 kB
Committed_AS: 143945284 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 1104960 kB
VmallocChunk: 33888896848 kB
HardwareCorrupted: 0 kB
AnonHugePages: 48400384 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 119168 kB
DirectMap2M: 11397120 kB
DirectMap1G: 525336576 kB

---

Dump from /proc/self/status

Name: mira
State: R (running)
Tgid: 6213
Pid: 6213
PPid: 6212
TracerPid: 0
Uid: 683 683 683 683
Gid: 504 504 504 504
FDSize: 64
Groups: 504 20200
VmPeak: 143776436 kB
VmSize: 143776428 kB
VmLck: 0 kB
VmHWM: 140154760 kB
VmRSS: 139583732 kB
VmData: 143769024 kB
VmStk: 136 kB
VmExe: 5792 kB
VmLib: 0 kB
VmPTE: 179376 kB
VmSwap: 581004 kB
Threads: 1
SigQ: 0/4135516
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000000000000
SigCgt: 0000000180000000
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: ffffffffffffffff
Cpus_allowed: ffffffff,ffffffff
Cpus_allowed_list: 0-63
Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,000
00000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,0000
0000,000000ff
Mems_allowed_list: 0-7
voluntary_ctxt_switches: 28998631
nonvoluntary_ctxt_switches: 13096904

---

Information on current assembly object:

AS_readpool: 82551522 reads.
AS_contigs: 0 contigs.
AS_bbcontigs: 0 contigs.
Mem used for reads: 184 (184 B)

Memory used in assembly structures:
Eff. Size Free cap. LostByAlign
AS_writtenskimhitsperid: 82551522 315 MiB 0 B 0 B
AS_skim_edges: 159528026 16.8 GiB 12.0 GiB 0 B
AS_adsfacts: 0 24 B 0 B 0 B
AS_confirmed_edges: 0 24 B 0 B 0 B
AS_permanent_overlap_bans: 1 24 B 0 B 0 B
AS_readhitmiss: 0 24 B 0 B 0 B
AS_readhmcovered: 0 24 B 0 B 0 B
AS_count_rhm: 0 24 B 0 B 0 B
AS_clipleft: 82551522 315 MiB 0 B 0 B
AS_clipright: 82551522 315 MiB 0 B 0 B
AS_used_ids: 82551522 79 MiB 0 B 6 B
AS_multicopies: 82551522 79 MiB 0 B 6 B
AS_hasmcoverlaps: 0 79 MiB 79 MiB 6 B
AS_maxcoveragereached: 82551522 315 MiB 0 B 0 B
AS_coverageperseqtype: 0 24 B 0 B 0 B
AS_istroublemaker: 82551522 79 MiB 0 B 6 B
AS_isdebris: 82551522 79 MiB 0 B 6 B
AS_needalloverlaps: 82551522 79 MiB 0 B 6 B
AS_readsforrepeatresolve: 0 40 B 0 B 0 B
AS_allrmbsok: 0 315 MiB 315 MiB 0 B
AS_probablermbsnotok: 0 315 MiB 315 MiB 0 B
AS_weakrmbsnotok: 0 315 MiB 315 MiB 0 B
AS_readmaytakeskim: 82551522 79 MiB 30 B 0 B
AS_skimstaken: 18413131972 17.1 GiB 60 B 0 B
AS_numskimoverlaps: 82551522 315 MiB 0 B 0 B
AS_numleftextendskims: 82551522 315 MiB 0 B 0 B
AS_rightextendskims: 82551522 315 MiB 0 B 0 B
AS_skimleftextendratio: 82551522 79 MiB 0 B 6 B
AS_skimrightextendratio: 82551522 79 MiB 0 B 6 B
AS_usedtmpfiles: 12 400 B 0 B 0 B
Total: 40497180000 (37.7 GiB)

================================================================================
Dynamic s allocs: 0
Dynamic m allocs: 0
Align allocs: 0

Internal logic/programming/debugging error (sigh this should not have
happened)

---

• More than 2^31 skim hits would be taken ... this was not foreseen yet. *
• Please contact the author. *

---

->Thrown: void Assembly::reduceSkimHits4(int32 version, const string prefix, const string postfix, const string logname)
->Caught: main

Aborting process, probably due to an internal error.

If you want to report the error, please do so on
http://sourceforge.net/p/mira-assembler/tickets/
and also give a short notice on the mira talk mailing list.

If reporting, please do not delete the log and checkpoint directories, there may
be files in them which could be needed to find the problem.
Subscribing / unsubscribing to mira talk, see: http://www.freelists.org/list/mira_talk

CWD: /scratch/anna/MIRA
Thank you for noticing that this is NOT a crash, but a
controlled program stop.
Failure, wrapped MIRA process aborted.

Do you have any suggestions what I can do or which parameters I should use
to obtain an accurate assembly?
In the attachment you will find the complete logfile.

Thank you in advance!

Best regards,
Anna

[Bastien Chevreux]

Terribly sorry, the data you are using is simply too big for MIRA4. Won't fix.