Huge memory used in intermed files

[Orson]

Dear team ksnp3

I chosse to use these program KNSP3 because I need find SNP of 1550 genomes. I have experience using parsnp but with it is not possible analized my data.

PD: I use a machine with 30 cores, 128 RAM. 950GB of hard disk.

I have few question: My input files are draft genome assemblys,. not reads. 5Mb each file.

1. I run ksnp3 and I can see that it generated a lot of intermed files, for my data was more of 900GB of intermed files. Exist a option for remove the intermed files that the program not use ?
2. I can see that anny steps not work using all cores that I have. I use the option -CPU 25. For example in the step Removing kmers that occur less than freq=average of median and mean kmer frequency for that genome. It work one by one file. Is possible run these steps using CPUs ?
3. How many space in hard disk you think that I need for finish my analysis. ?

Finally:

**I have these error running the program: I think that is only problem with space in hard disk. Or maybe RAM memory. What is your opinion ? **

Concatenate results for each genome and sort by locus to create SNPs_all_labelLoci
Thu Oct 11 11:13:36 UTC 2018
.
.
.
genome: 790-97_Peru_2007_C in Dir.fsplit99
genome: GCA_002221085_1_ASM222108v1_2013_NA_C in Dir.fsplit990
genome: GCA_002221095_1_ASM222109v1_2013_New_England_C in Dir.fsplit991
genome: GCA_002221145_1_ASM222114v1_2013_NA_C in Dir.fsplit992
genome: GCA_002221165_1_ASM222116v1_2013_NA_C in Dir.fsplit993
genome: GCA_002221175_1_ASM222117v1_2013_NA_C in Dir.fsplit994
genome: GCA_002221185_1_ASM222118v1_2013_NA_C in Dir.fsplit995
genome: GCA_002221225_1_ASM222122v1_2013_NA_E in Dir.fsplit996
genome: GCA_002221245_1_ASM222124v1_2013_NA_E in Dir.fsplit997
genome: GCA_002221265_1_ASM222126v1_2013_NA_C in Dir.fsplit998
genome: GCA_002221285_1_ASM222128v1_2014_NA_C in Dir.fsplit999
sort: write failed: /tmp/sort5v6kCZ: No space left on device
Number_SNPs: 1
$count_snps: 0
Finished finding SNPs
Thu Oct 11 12:36:46 UTC 2018
rm: cannot remove 'TemporaryFilesToDelete': No such file or directory
mv: No match.

I hope your help. Thank you so much.

Orson.

Saludos

[Iowa]

I have the same proglem, ~1300 genomes, more than 800GB of data generated on hard drive.

Is there a solution?

SOLUTION: I used a dedicated high capacity SSD

I ran into a second problem:

Now it runs smoothly, jellyfish runs but then there is a step where it has to write into dedicated directoryies. It runs ok the first 600 genomes than it reports for every other genomes
"awk: write failure (File too large) awk: close failed on file /dev/stdout (File too large)"
until the last one.

It generates a fasta matrix of ~600 sequences and then it strats building trees.
At these stage it plots "TOO FEW SPECIES" when doing the core stage.

I have already used ksnp several times in the past and recently, but this is the first time I encounter such problems.

INFO: All genomes are from the same species; input is, for every genome, the final assembly with multifasta containing multiple contigs generated from the assembly (~5MB files); machine is 32 core 64GB of RAM 2x1TB SSD.

**SOLUTION: I have checked all my assemblies and some were not good quality, removing/correcting them made me finish the work with good results. It still uses nearly 1TB of disk space and still is not able to delete automatically the folder "TemporaryFilesToDelete". **

Thanks for the support provided.