annotate_fusions.pl out of memory!

[eearley]

All of my runs fail at the same step: annotate_fusions.pl

From the end of defuse_out_dir/log/defuse.log:

Starting defuse command:
/opt/pkg-big/defuse-0.6.1/scripts/annotate_fusions.pl -c /opt/pkg-big/defuse-0.6.1/config.txt -o ~/test_subsample_Mix/Mix_500K_defuse -n Mix_500K_defuse > #>1
Reasons:
~/test_subsample_Mix/Mix_500K_defuse/annotations missing
Failure for defuse command:
/opt/pkg-big/defuse-0.6.1/scripts/annotate_fusions.pl -c /opt/pkg-big/defuse-0.6.1/config.txt -o ~/test_subsample_Mix/Mix_500K_defuse -n Mix_500K_defuse > ~/test_subsample_Mix/Mix_500K_defuse/annotations.tmp
Reason:
Job command with nonzero return code
Return codes: 1
Job output:
Running on ea-bio-c07
Out of memory!

    real    3m12.931s
    user    3m3.979s
    sys     0m6.968s

1 commands failed after 194 seconds

---

I've ruled out memory allocation issues on my end (ie, 64gb RAM available to me), and I've tried running the entire defuse package on downsampled FASTQs (1million PE reads, 500K, 100K). Every instance fails the same way.

I also tracked actual resource use on the annotate_fusions.pl itself:

/usr/bin/time -v perl /opt/pkg-big/defuse-0.6.1/scripts/annotate_fusions.pl -c /opt/pkg-big/defuse-0.6.1/config.txt -o ~/test_subsample_Mix/Mix_500K_defuse -n Mix_500K_defuse

Out of memory!
Command exited with non-zero status 1
Command being timed: "perl /opt/pkg-big/defuse-0.6.1/scripts/annotate_fusions.pl -c /opt/pkg-big/defuse-0.6.1/config.txt -o ~/test_subsample_Mix/Mix_500K_defuse -n Mix_500K_defuse"
User time (seconds): 161.16
System time (seconds): 4.28
Percent of CPU this job got: 99%
Elapsed (wall clock) time (h:mm:ss or m:ss): 2:45.60
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 20803616
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 1317200
Voluntary context switches: 197
Involuntary context switches: 14102
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 1

---

Again, without fail, the script hits 20gb memory then quits. Is there some job control parameter that sets 20gb max memory somewhere? Or any other ideas?

Thanks!

[eearley]

update:
I managed to finish annotation on a separate cluster, so I do actually suspect this was a non-defuse issue.

Does annotate_fusions tend to spike sharply in mem use at the tail end? When tracking in top, I notice the process crashes at ~7gb RAM, yet resource use according to time reports 20gb RAM. Maybe this is a cluster issue, too...

[Andrew]

Apologies for this. Yes annotate_fusions.pl is not well optimized in terms of memory. Note that annotate_fusions.pl also calls out to other scripts/executables that are somewhat memory intensive, especially if you have calculate_extra_annotations set to yes.