From: Robert D. <rm...@sa...> - 2022-08-18 14:17:46
|
Samtools (and HTSlib and BCFtools) version 1.16 is now available from GitHub and SourceForge. https://sourceforge.net/projects/samtools/ https://github.com/samtools/htslib/releases/tag/1.16 https://github.com/samtools/samtools/releases/tag/1.16 https://github.com/samtools/bcftools/releases/tag/1.16 The main changes are listed below: ------------------------------------------------------------------------------ htslib - changes v1.16 ------------------------------------------------------------------------------ * Make hfile_s3 refresh AWS credentials on expiry in order to make HTSlib work better with AWS IAM credentials, which have a limited lifespan. (PR#1462 and PR#1474, addresses #344) * Allow BAM headers between 2GB and 4GB in size once more. This is not permitted in the BAM specification but was allowed in an earlier version of HTSlib. There is now a warning at 2GB and a hard failure at 4GB. (PR#1421, fixes #1420 and samtools#1613. Reported by John Marshall and R C Mueller) * Improve error message when failing to load an index. (PR#1468, example of the problem samtools#1637) * Permit MM (base modification) tags containing "." and "?" suffixes. These define implicit vs explicit coordinates. See the SAM tags specification for details. (PR#1423 and PR#1426, fixes #1418. PR#1469, fixes #1466. Reported by cjw85) * Warn if spaces instead of tabs are detected in a VCF file to prevent confusion. (PR#1328, fixes bcftools#1575. Reported by ketkijoshi278) * Add an "sclen" filter expression keyword. This is the length of a soft-clip, both left and right end. It may be combined with qlen (qlen-sclen) to obtain the number of bases in the query sequence that have been aligned to the genome ie it provides a way to compare local-alignment vs global-alignment length. (PR#1441 and PR/samtools#1661, fixes #1436. Requested by Chang Y) * Improve error messages for CRAM reference mismatches. If the user specifies the wrong reference, the CRAM slice header MD5sum checks fail. We now report the SQ line M5 string too so it is possible to validate against the whole chr in the ref.fa file. The error message has also been improved to report the reference name instead of #num. Finally, we now hint at the likely cause, which counters the misleading samtools supplied error of "truncated or corrupt" file. (PR#1427, fixes samtools#1640. Reported by Jian-Guo Zhou) * Expose more of the CRAM API and add new functionality to extract the reference from a CRAM file. (PR#1429 and PR#1442) * Improvements to the implementation of embedded references in CRAM where no external reference is specified. (PR#1449, addresses some of the issues in #1445) * The CRAM writer now allows alignment records with RG:Z: aux tags that don't have a corresponding @RG ID in the file header. Previously these tags would have been silently dropped. HTSlib will complain whenever it has to add one though, as such tags do not conform to recommended practice for the SAM, BAM and CRAM formats. (PR#1480, fixes #1479. Reported by Alex Leonard) * Set tab delimiter in man page for tabix GFF3 sort. (PR#1457. Thanks to Colin Diesh) * When using libdeflate, the 1...9 scale of BGZF compression levels is now remapped to the 1...12 range used by libdeflate instead of being passed directly. In particular, HTSlib levels 8 and 9 now map to libdeflate levels 10 and 12, so it is possible to select the highest (but slowest) compression offered by libdeflate. (PR#1488, fixes #1477. Reported by Gert Hulselmans) * The VCF variant API has been extended so that it can return separate flags for INS and DEL variants as well as the existing INDEL one. These flags have not been added to the old bcf_get_variant_types() interface as it could break existing users. To access them, it is necessary to use new functions bcf_has_variant_type() and bcf_has_variant_types(). (PR#1467) * The missing, but trivial, `le_to_u8()` function has been added to hts_endian. (PR#1494, Thanks to John Marshall) * bcf_format_gt() now works properly on big-endian platforms. (PR#1495, Thanks to John Marshall) Build changes ------------- These are compiler, configuration and makefile based changes. * Update htscodecs to version 1.3.0 for new SIMD code + various fixes. Updates the htscodecs submodule and adds changes necessary to make HTSlib build the new SIMD codec implementations. (PR#1438, PR#1489, PR#1500) * Fix clang builds under mingw. Under mingw, clang requires dllexport to be applied to both function declarations and function definitions. (PR#1435, PR#1497, PR#1498 fixes #1433. Reported by teepean) * Fix curl type warning with gcc 12.1 on Windows. (PR#1443) * Detect ARM Neon support and only build appropriate SIMD object files. (PR#1451, fixes #1450. Thanks to John Marshall) * `make print-config` now reports extra CFLAGS that are needed to build the SIMD parts of htscodecs. These may be of use to third-party build systems that don't use HTSlib's or htscodecs' build infrastructure. (PR#1485. Thanks to John Marshall) * Fixed some Makefile dependency issues for the "check"/"test" targets and plugins. In particular, "make check" will now build the "all" target, if not done already, before running the tests. (PR#1496) Bug fixes --------- * Fix bug when reading position -1 in BCF (0 in VCF), which is used to indicate telomeric regions. The BCF reader was incorrectly assuming the value stored in the file was unsigned, so a VCF->BCF->VCF round-trip would change it from 0 to 4294967296. (PR#1476, fixes #1475 and bcftools#1753. Reported by Rodrigo Martin) * Various bugs and quirks have been fixed in the filter expression engine, mostly related to the handling of absent tags, and the is_true flag. Note that as a result of these fixes, some filter expressions may give different results: - Fixed and-expressions including aux tag values which could give an invalid true result depending on the order of terms. - The expression `![NM]` is now true if only `NM` does not exist. In earlier versions it would also report true for tags like `NM:i:0` which exist but have a value of zero. - The expression `[X1] != 0` is now false when `X1` does not exist. Earlier versions would return true for this comparison when the tag was missing. - NULL values due to missing tags now propagate through string, bitwise and mathematical operations. Logical operations always treat them as false. (PR#1463, fixes samtools#1670. Reported by Gert Hulselmans; PR#1478, fixes samtools#1677. Reported by johnsonzcode) * Fix buffer overrun in bam_plp_insertion_mod. Memory now grows to the proper size needed for base modification data. (PR#1430, fixes samtools#1652. Reported by hd2326) * Remove limit of returned size from fai_retrieve(). (PR#1446, fixes samtools#1660. Reported by Shane McCarthy) * Cap hts_getline() return value at INT_MAX. Prevents hts_getline() from returning a negative number (a fail) for very long string length values. (PR#1448. Thanks to John Marshall) * Fix breakend detection and test bcf_set_variant_type(). (PR#1456, fixes #1455. Thanks to Martin Pollard) * Prevent arrays of BCF_BT_NULL values found in BCF files from causing bcf_fmt_array() to call exit() as the type is unsupported. These are now tested for and caught by bcf_record_check(), which returns an error code instead. (PR#1486) * Improved detection of fasta and fastq files that have very long comments following identifiers. (PR#1491, thanks to John Marshall. Fixes samtools/samtools#1689, reported by cjw85) * Fixed a SEGV triggered by giving a SAM file to `samtools import`. (PR#1492) ------------------------------------------------------------------------------ samtools - changes v1.16 ------------------------------------------------------------------------------ New work and changes: * samtools reference command added. This subcommand extracts the embedded reference out of a CRAM file. (PR#1649, addresses #723. Requested by Torsten Seemann) * samtools import now adds grouped by query-name to the header. (PR#1633, thanks to Nils Homer) * Made samtools view read error messages more generic. Former error message would claim that there was a "truncated file or corrupt BAM index file" with no real justification. Also reset errno in stream_view which could lead to confusing error messages. (PR#1645, addresses some of the issues in #1640. Reported by Jian-Guo Zhou) * Make samtools view -p also clear mqual, tlen and cigar. (PR#1647, fixes #1606. Reported by eboyden) * Add bedcov option -c to report read count. (PR#1644, fixes #1629. Reported by Natchaphon Rajudom) * Add UMI/barcode handling to samtools markdup. (PR#1630, fixes #1358 and #1514. Reported by Gert Hulselmans and Poshi) * Add a new template coordinate sort order to samtools sort and samtools merge. This is useful when working with unique molecular identifiers (UMIs). (PR#1605, fixes #1591. Thanks to Nils Homer) * Rename mpileup --ignore-overlaps to --ignore-overlaps-removal or --disable-overlap-removal. The previous name was ambiguous and was often read as an option to enable removal of overlapping bases, while in reality this is on by default and the option turns off the ability to remove overlapping bases. (PR#1666, fixes #1663. Reported by yangdingyangding) * The dict command can now read BWA's .alt file and add AH:* tags indicating reference sequences that represent alternate loci. (PR#1676. Thanks to John Marshall) * The "samtools index" command can now accept multiple alignment filenames with the new -M option, and will index each of them separately. (Specifying the output index filename via out.index or the new -o option is currently only applicable when there is only one alignment file to be indexed.) (PR#1674. Reported by Abigail Ramsøe and Nicola Romanò. Thanks to John Marshall) * Allow samtools fastq -T "*". This allows all tags from SAM records to be written to fastq headers. This is a counterpart to samtools import -T "*". (PR#1679. Thanks to cjw85) Bug Fixes: * Re-enable --reference option for samtools depth. The reference is not used but this makes the command line usage compatible with older releases. (PR#1646, fixes #1643. Reported by Randy Harr) * Fix regex coordinate bug in samtools markdup. (PR#1657, fixes #1642. Reported by Randy Harr) * Fix divide by zero in plot-bamstats -m, on unmapped data. (PR#1678, fixes #1675. Thanks to Shane McCarthy) * Fix missing RG headers when using samtools merge -r. (PR#1683, addresses htslib#1479. Reported by Alex Leonard) * Fix a possible unaligned access in samtools reference. (PR#1696) Documentation: * Add documentation on CRAM compression profiles and some of the newer options that appear in CRAM 3.1 and above. (PR#1659, fixes #1656. Reported by Matthias De Smet) * Add "sclen" filter expression keyword documentation. (PR#1661, see also htslib#1441) * Extend FILTER EXPRESSION man page section to match the changes made in HTSlib. (PR#1687, samtools/htslib#1478) Non user-visible changes and build improvements: * Ensure generated test files are ignored (by git) and cleaned (by make testclean) (PR#1692, Thanks to John Marshall) ------------------------------------------------------------------------------ bcftools - changes v1.16 ------------------------------------------------------------------------------ * New plugin `bcftools +variant-distance` to annotate records with distance to the nearest variant (#1690) Changes affecting the whole of bcftools, or multiple commands: * The -i/-e filtering expressions - Added support for querying of multiple filters, for example `-i 'FILTER="A;B"'` can be used to select sites with two filters "A" and "B" set. See the documentation for more examples. - Added modulo arithmetic operator Changes affecting specific commands: * bcftools annotate - A bug introduced in 1.14 caused that records with INFO/END annotation would incorrectly trigger `-c ~INFO/END` mode of comparison even when not explicitly requested, which would result in not transferring the annotation from a tab-delimited file (#1733) * bcftools merge - New `-m snp-ins-del` switch to merge SNVs, insertions and deletions separately (#1704) * bcftools mpileup - New NMBZ annotation for Mann-Whitney U-z test on number of mismatches within supporting reads - Suppress the output of MQSBZ and FS annotations in absence of alternate allele * bcftools +scatter - Fix erroneous addition of duplicate PG lines * bcftools +setGT - Custom genotypes (e.g. `-n c:1/1`) now correctly override ploidy -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. |