[Samtools-devel] Release 1.19

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

Samtools (and HTSlib and BCFtools) version 1.19 is now available from
GitHub and SourceForge.

https://github.com/samtools/htslib/releases/tag/1.19
https://github.com/samtools/samtools/releases/tag/1.19
https://github.com/samtools/bcftools/releases/tag/1.19
https://sourceforge.net/projects/samtools/

The main changes are listed below:

------------------------------------------------------------------------------
htslib - changes v1.19
------------------------------------------------------------------------------

Updates
-------

* A temporary work-around has been put in the VCF parser so that it is less
   likely to fail on rows with a large number of ALT alleles, where Number=G
   tags like PL can expand beyond the 2Gb limit enforced by HTSlib.  For now,
   where this happens the offending tag will be dropped so the data can be
   processed, albeit without the likelihood data.

   In future work, the library will instead convert such tags into their
   local alternatives (see https://github.com/samtools/hts-specs/pull/434).

* New program. Adds annot-tsv which annotates regions in a destination file
   with texts from overlapping regions in a source file. (PR#1619)

* Change bam_parse_cigar() so that it can modify existing BAM records.  This
   makes more useful as public API.  Previously it could only handle partially
   formed BAM records. (PR#1651, fixes #1650. Reported by Oleksii Nikolaienko)

* Add "uncompressed" to hts_format_description() where appropriate.  This
   adds an "uncompressed" description to uncompressed files that would
   normally be compressed, such as BAM and BCF. (PR#1656, in relation to
   samtools#1884.  Thanks to John Marshall)

* Speed up to the VCF parser and writer. (PR#1644 and PR#1663)

* Add an hclen (hard clip length) SAM filter function. (PR#1660, with
   reference to samtools#813)

* Avoid really closing stdin/stdout in hclose()/hts_close()/et al. See
   discussion in PR for details. (PR#1665.  Thanks to John Marshall)

* Add support to handle multiple files in bgzip. (PR#1658, fixes #1642.
   Requested by bw2)

* Enable auto-vectorisation in CRAM 3.1 codecs.  Speeds decoding on some
   sequencing platform data. (PR#1669)

* Speed up removal of lines in large headers. (PR#1662, fixes #1460.
   Reported by Anže Starič)

* Apply seqtk PR to improve kseq.h parsing performance.  Port of
   Fabian Klötzl's (kloetzl) lh3/seqtk#123 and attractivechaos/klib#173
   to HTSlib. (PR#1674.  Thanks to John Marshall)

Build changes
-------------

* Updated htscodecs submodule to 1.6.0. (PR#1685, PR#1717, PR#1719)

* Apply the packed attribute to uint*_u types for Clang to prevent
   -fsanitize=alignment failures. (PR#1667.  Thanks to Fangrui Song)

* Fuzz testing improvements. (PR#1664)

* Add C++ casts for external headers in klist.h and kseq.h. (PR#1683.  See
   also PR#1674 and PR#1682)

* Add test case compiling the public headers as C++. (PR#1682.  Thanks to
   John Marshall)

* Enable optimisation level -O3 for SAM QUAL+33 formatting. (PR#1679)

* Make compiler flag detection work with zig cc. (PR#1687)

* Fix unused value warnings when built with NDEBUG. (PR#1688)

* Remove some disused Makefile variables, fix typos and a warning.  Improve
   bam_parse_basemod() documentation. (PR#1705, Thanks to John Marshall)

Bug fixes
---------

* Fail bgzf_useek() when offset is above block limits. (PR#1668)

* Fix multi-threaded on-the-fly indexing problems. (PR#1672, fixes
   samtools#1861 and bcftools#1985.  Reported by Mark Ebbert and lacek)

* Fix hfile_libcurl small seek bug. (PR#1676, fixes samtools#1918.  Also may
   fix #1037, #1625 and samtools#1622. Reported by Alex Reynolds, Mark Walker,
   Arthur Gilly and skatragadda-nygc. Thanks to John Marshall)

* Fix a minor memory leak in malformed CRAM EXTERNAL blocks. [fuzz] (PR#1671)

* Fix a cram decode hang from block_resize(). (PR#1680. Reported by
   Sebastian Deorowicz)

* Cram fuzzing improvements.  Fixes a number of cram errors. (PR#1701,
   fixes #1691, #1692, #1693, #1696, #1697, #1698, #1699 and #1700. Thanks
   to Octavio Galland for finding and reporting all these)

* Fix crypt4gh redirection. (PR#1675, fixes  grbot/crypt4gh-tutorial#2.
   Reported by hth4)

* Fix PG header linking when records make a loop. (PR#1702, fixes #1694.
   Reported by Octavio Galland)

* Prevent issues with no-stored-sequence records in CRAM files, by ensuring
   they are accounted for properly in block size calculations, and by limiting
   the maximum query length in the CIGAR data.  Originally seen as an overflow
   by OSS-Fuzz / UBSAN, it turned out this could lead to excessive time and
   memory use by HTSlib, and could result in it writing out unreadable CRAM
   files. (PR#1710)

* Fix some illegal shifts and integer overflows found by OSS-Fuzz / UBSAN.
   (PR#1707, PR#1712, PR#1713)

------------------------------------------------------------------------------
samtools - changes v1.19
------------------------------------------------------------------------------

New work and changes:

* Samtools coverage: add a new --plot-depth option to draw depth (of
   coverage) rather than the percentage of bases covered. (PR #1910.
   Thanks to Pierre Lindenbaum)

* Samtools merge / sort: add a lexicographical name-sort option via the -N
   option.  The "natural" alpha-numeric sort is still available via -n.
   (PR #1900, fixes #1500.  Reported by Steve Huang)

* Samtools view: add -N ^NAME_FILE and -R ^RG_FILE options.  The standard -N
   and -R options only output reads matching a specified list of names or
   read-groups.  With a caret (^) prefix these may be negated to only output
   read not matching the specified files. (PR #1896, fixes #1895.  Suggested
   by Feng Tian)

* Cope with Htslib's change to no longer close stdout on hts_close. Htslib
   companion PR is samtools/htslib#1665. (PR #1909.  Thanks to John Marshall)

* Plot-bamstats: add a new plot of the read lengths ("RL") from samtools
   stats output. (PR #1922, fixes #1824.  Thanks to @erboone, suggested by
   Martin Pollard)

* Samtools split: support splitting files based on the contents of auxiliary
   tags.  Also adds a -M option to limit the number of files split can make,
   to avoid accidental resource over-allocation, and fixes some issues with
   --write-index. (PR #1222, PR #1933, fixes #1758.  Thanks to Valeriu Ohan,
   suggested by Scott Norton)

Bug Fixes:

* Samtools stats: empty barcode tags are now treated as having no barcode.
   (PR #1929, fixes #1926.  Reported by Jukka Matilainen)

* Samtools cat: add support for non-seekable streams.  The file format
   detection blocked pipes from working before, but now files may be
   non-seekable such as stdin or a pipe. (PR #1930, fixes #1731.  Reported
   by Julian Hess)

* Samtools mpileup -aa (absolutely all positions) now produces an output even
   when given an empty input file. (PR #1939.  Reported by Chang Y)

* Samtools markdup: speed up optical duplicate tagging on regions with very
   deep data. (PR #1952)

Documentation:

* Samtools mpileup: add more usage examples to the man page. (PR #1913,
   fixes #1801)

* Samtools fastq: explicitly document the order that filters apply.
   (PR #1907)

* Samtools merge: fix example output to use an uppercase RG PL field.
   (PR #1917.  Thanks to John Marshall.  Reported by Michael Macias)

* Add hclen SAM filter documentation. (PR #1902.  See also
   samtools/htslib#1660)

* Samtools consensus: remove the -5 option from documentation.  This option
   was renamed before the consensus subcommand was merged, but accidentally
   left in the man page. (PR #1924)

------------------------------------------------------------------------------
bcftools - changes v1.19
------------------------------------------------------------------------------

Changes affecting the whole of bcftools, or multiple commands:

* Filtering expressions can be given a file with list of strings to match,
   this was previously possible only for the ID column. For example

   ID=@file            .. selects lines with ID present in the file
   INFO/TA...@fi...  .. selects lines where TAG has a string value listed
                          in the file
   INFO/TAG!=@file.txt .. TAG must not have a string value
                          listed in the file

* Allow to query REF,ALT columns directly, for example

   -e 'REF="N"'

Changes affecting specific commands:

* bcftools annotate

     - Fix `bcftools annotate --mark-sites`, VCF sites overlapping regions in
       a BED file were not annotated (#1989)

     - Add flexibility to FILTER column transfers and allow transfers within
       the same file, across files, and in combination. For examples see
       http://samtools.github.io/bcftools/howtos/annotate.html#transfer_filter_to_info

* bcftools call

     - Output MIN_DP rather than MinDP in gVCF mode

     - New `-*, --keep-unseen-allele` option to output the unobserved
       allele <*>, intended for gVCF.

* bcftools head

     - New `-s, --samples` option to include the #CHROM header line with
       samples.

* bcftools gtcheck

     - Add output options `-o, --output` and `-O, --output-type`

     - Add filtering options `-i, --include` and `-e, --exclude`

     - Rename the short option `-e, --error-probability` from lower case to
       upper case `-E, --error-probability`

     - Changes to the output format, replace the DC section with DCv2:

         - adds a new column for the number of matching genotypes

         - The --error-probability is newly interpreted as the probability of
           erroneous allele rather than genotype. In other words, the
           calculation of the discordance score now considers the probability
           of genotyping error to be different for HOM and HET genotypes,
           i.e. P(0/1|dsg=0) > P(1/1|dsg=0).

         - fixes in HWE score calculation plus output average HWE score rather
           than absolute HWE score

         - better description of fields

* bcftools merge

     - Add `-m` modifiers to suppress the output of the unseen allele <*> or
       <NON_REF> at variant sites (e.g. `-m both,*`) or all sites
       (e.g. `-m both,**`)

* bcftools mpileup

     - Output MIN_DP rather than MinDP in gVCF mode

* bcftools norm

     - Add the number of joined lines to the summary output, for example

       Lines   total/split/joined/realigned/skipped:  6/0/3/0/0

     - Allow combining -m and -a with --old-rec-tag (#2020)

     - Symbolic <DEL> alleles caused norm to expand REF to the full length of
       the deletion. This was not intended and problematic for long deletions,
       the REF allele should list one base only (#2029)

* bcftools query

     - Add new `-N, --disable-automatic-newline` option for pre-1.18 query
       formatting behavior when newline would not be added when missing

     - Make the automatic addition of the newline character in a more
       predictable way and, when missing, always put it at the end of the
       expression. In version 1.18 it could be added at the end of the
       expression (for per-site expressions) or inside the square brackets
       (for per-sample expressions). The new behavior is:

         - if the formatting expression contains a newline character, do
           nothing

         - if there is no newline character and -N,
           --disable-automatic-newline is given, do nothing

         - if there is no newline character and -N is not given, insert
           newline at the end of the expression

  See #1969 for details

     - Add new `-F, --print-filtered` option to output a default string for
       samples that would otherwise be filtered by `-i/-e` expressions.

     - Include sample name in the output header with `-H` whenever it makes
       sense (#1992)

* bcftools +spit-vep

     - Fix on the fly filtering involving numeric subfields, e.g. `-i
       'MAX_AF<0.001'` (#2039)

     - Interpret default column type names (--columns-types) as entire
       strings, rather than substrings to avoid unexpected spurious
       matches (i.e. internally add ^ and $ to all field names)

* bcftools +trio-dnm2

     - Do not flag paternal genotyping errors as de novo mutations.
       Specifically, when father's chrX genotype is 0/1 and mother's 0/0,
       0/1 in the child will not be marked as DNM.

* bcftools view

     - Add new `-A, --trim-unseen-allele` option to remove the unseen allele
       <*> or <NON_REF> at variant sites (`-A`) or all sites (`-AA`)

-- 
 The Wellcome Sanger Institute is operated by Genome Research 
 Limited, a charity registered in England with number 1021457 and a 
 company registered in England with number 2742969, whose registered 
 office is Wellcome Sanger Institute, Wellcome Genome Campus, 
 Hinxton, CB10 1SA.