#24 filterbytile.sh successive run results are (slightly) different

[Anandkumar Surendrarao]

Hi Brian,

I am using BBMap version 38-60 and for the filterbytile.sh step, I ran it successively a few times, but each time the results are slightly different, as shown in my pairwise comparison example below:

   2392008 SRR1726469_renamed_OptiDeDupe_FbT_thread1_manual.fastq.gz
   2390262 SRR1726469_renamed_OptiDeDupe_FbT_thread1_manual_repeat.fastq.gz

. The syntax I am using is the generic:
filterbytile.sh -Xmx40g in=$IN.fastq.gz out=$OUT.fastq.gz threads=1

Results still fluctuate with automatic detection of number of threads as well, using following syntax:
filterbytile.sh -Xmx40g in=$IN.fastq.gz out=$OUT.fastq.gz threads=auto

Just like for clumpify.sh, variability across runs could be fixed using the same positve integer as seed, e.g. seed=1, is it possible to use this same trick to get reproducible results for filterbytile.sh as well?

It does not appear so, and therefore I am pinging you for help, here on this ticket submission page, that Genomax pointed out at Biostars.

Thank you,
Anand

[Anandkumar Surendrarao]

UPDATE:
I just found that using the following flag allows me to obtain reproducible results with: usekmers=f

This syntax idea arose from an old seqanswers post by you at this link

Disabling kmer uniqueness to increase speed and decrease memory usage:
Code:
filterbytile.sh in=x.fq out=y.fq usekmers=f

However, in the help menu for filterbytile.sh, version 38-60, it says -

A micro-tile is discarded if any of 3 metrics indicate a problem. The metrics are kmer uniqueness (u), average quality (q), and probability of being error-free (e). Each has 3 parameters: deviations (d), fraction (f), and absolute (a). After calculating the difference (delta) between a micro-tile and average, it is discarded only if all three of these conditions are true for at least one metric (using quality as the example): 1) delta is greater than (qd) standard deviations. 2) delta is greater than average times the fraction (qf). 3) delta is greater than the absolute value (qa).

Therefore, I wonder under what circumstances using only average quality (q), and probability of being error-free (e), and excluding kmer uniqueness (u) would be OK versus not OK.... Please share your thoughts / experience. Thanks!

This response is cross-posted with https://www.biostars.org/p/403347/#404029. Thanks, Brian!

[Brian Bushnell]

I added a "seed" flag for the next release (30.71), which should make the program deterministic. The default seed is -1, and negative seeds will still produce nondeterminstic output. Positive seeds will set the random number generator into a specific state and produce deterministic output as long as the same seed is used.

[Anandkumar Surendrarao]

Thank you, Brian

[Anandkumar Surendrarao]

Deleting here and re-posting as a new question on filterbytile 'group' selection for stats calculation. Thanks!