Menu
▾
▴
[Samtools-announce] Release 1.14
|
From: Robert D. <rm...@sa...> - 2021-10-22 15:08:27
|
Samtools (and HTSlib and BCFtools) version 1.14 is now available from GitHub and SourceForge. https://sourceforge.net/projects/samtools/ https://github.com/samtools/htslib/releases/tag/1.14 https://github.com/samtools/samtools/releases/tag/1.14 https://github.com/samtools/bcftools/releases/tag/1.14 The main changes are listed below: ------------------------------------------------------------------------------ htslib - changes v1.14 ------------------------------------------------------------------------------ Features and Updates -------------------- * Added a keep option to bgzip to leave the original file untouched. This brings bgzip into line with gzip. (PR #1331, thanks to Alex Petty) * "endpos" has been added to the filter language, giving the position of the rightmost mapped base as measured by the CIGAR string. For unmapped reads it is the same as "pos". (PR #1307, thanks to John Marshall) * Interfaces have been added to interpret the new base modification tags added to the SAMtags document in samtools/hts-specs#418. (PR #1132) * New API functions hts_flush()/sam_flush()/bcf_flush() for flushing output htsFile/samFile/vcfFile streams. (PR #1326, thanks to John Marshall) * The synced_bcf_reader now sorts lines with symbolic alleles by END tag as well as POS. (PR #1321) * Added synced_bcf_reader options BCF_SR_REGIONS_OVERLAP and BCF_SR_TARGETS_OVERLAP for better control of records that start outside the desired region but overlap it are handled. Fixes samtools/bcftools#1420 and samtools/bcftools#1421 raised by John Marshall. (PR #1327) * HTSlib will now accept long-cigar CG:B: tags made by htsjdk which don't quite follow the specification properly (using signed values instead of unsigned). Thanks to Colin Diesh for reporting an example file. (PR #1317) * The warning printed when the BGZF reader finds a file with no EOF block has been changed to be less alarming. Unfortunately some third-party BGZF encoders don't write EOF blocks at the end of files. Thanks to Keiran Raine for reporting an example file. (PR #1323) * The FASTA and FASTQ readers get an option to skip over the first item on the header line, and use the second as the read name. It allows the original name to be restored on some of the fastq files served from the European Nucleotide Archive (ENA). (PR #1325) * HTSlib is now more strict when parsing the VCF samples line (beginning #CHROM). It will only accept tabs between the mandatory field names and sample names must be separated with tabs. (PR #1328) * HTSlib will now warn if it looks like the header has been corrupted by diagnostic messages from the program that made it. This can happen when using `nohup`, which by default mixes stdout and stderr into the same stream. (PR#1339, thanks to John Marshall) * File format detection will now recognise signatures for XZ, Zstd and D4 files (note that HTSlib will not read them yet). (PR #1340, thanks to John Marshall) Build changes ------------- These are compiler, configuration and makefile based changes. * Some redundant tests have been removed from the test harness, speeding it up. (PR #1308) * The version.sh script now works better on shallow checkouts. (PR #1324) * A check-untracked Makefile target has been added to catch untracked files (mostly) left by the test harness. (PR #1324) Bug fixes --------- * Fixed a case where flushing the thread pool could very occasionally cause a deadlock. (PR #1309) * Fixed a bug where some CRAM files could fail to decode if the required_fields option was in use. Thanks to Matt Sexton for reporting the issue. (PR #1314, fixes samtools/samtools#1475) * Fixed a regression where the S3 plugin could not read public files unless you supplied some Amazon credentials. Thanks to Chris Saunders for reporting. (PR #1332, fixes samtools/samtools#1491) * Fixed a possible CRAM thread deadlock discovered by @ryancaicse. (PR #1330, fixes #1329) * Some set-but-unused variables have been removed. (PR #1334) * Fixed a bug which prevented "flag.read2" from working in the filter language unless it was at the end of the expression. Thanks to Vamsi Kodali for reporting the issue. (PR #1342) * Fixed a memory leak that could happen if CRAM fails to inflate a LZMA block. (PR #1340, thanks to John Marshall) ------------------------------------------------------------------------------ samtools - changes v1.14 ------------------------------------------------------------------------------ Notice: * Samtools mpileup VCF and BCF output (deprecated in release 1.9) will be removed in the next release. Please use bcftools mpileup instead. New work and changes: * The legacy samtools API (libbam.a, bam_endian.h, sam.h and most of bam.h) has been removed. We recommend coding against the HTSlib API directly. The legacy API had not been actively maintained since 2015. (#1483) * New "samtools samples" command to list the samples used in a SAM/BAM/CRAM file. (#1432; thanks to Pierre Lindenbaum) * "mpileup" now supports base modifications via the SAM Mm/MM auxiliary tag. Please see the "--output-mods" option. (#1311) * Added "mpileup --output-BP-5" option to output the BP field in 5' to 3' order instead of left to right. (#1484; fixes #1481) * Added "samtools view --rf" option as an additional FLAG filtering method. This keeps records only if (FLAG & N) != 0. (#1508; fixes #1470) * New "samtools import -N" option to use the second word on a FASTQ header line, matching the SRA/ENA FASTQ variant. (#1485) * Improve "view -x" option to simplify specifying multiple tags, and added the reverse "--keep-tag" option to include rather than exclude. (#516) * Switched the processing order of "view" -x (tag filtering) and -e (expression) handling. Expressions now happen first so we can filter on tags which are about to be deleted. This is now consistent with the "view -d" behaviour too. (#1480; fixes #1476. Reported by William Rowell) * Added filter expression "endpos" keyword. (#1464. Thanks to John Marshall) * "samtools view" errors now appear after any SAM output, improving their visibility. (#1490. Thanks to John Marshall) * Improved "samtools sort" use of temporary files, both tidying up if it fails and recovery when facing pre-existing temporary files. (#1510; fixes #1035, #1503. Reported by Vivek Rai and Maarten Kooyman) * Filtering in "samtools markdup" now sets the UNMAP BAM flag when given the "-p" option. (#1512; fixes #1469) * Make CRAM references shared during "samtools merge" so merging many files has a lower memory usage. (#471) Bug fixes: * Prevent "samtools depth" from closing stdout when outputting to terminal, avoiding a bad interaction with PySam. (#1465. Thanks to John Marshall) * In-place "samtools reheader" now works on CRAMs produced using a higher than default compression level. (#1479) * Fix setting of the dt tag in "markdup". Optical duplicates were being marked too early, negating the tagging and counting elsewhere. (#1487; fixes #1486. Reported by Kevin Lewis) * Reinstate the "samtools stats -I" option to filter by sample. (#1496; fixes #1489. Reported by Matthias Bernt) * Fix "samtools fastq" handling of dual index tags on single-ended input. (#1474) * Improve "samtools coverage" documentation. (#1521; fixes #1504. Reported by Peter Menzel) Non user-visible changes and build improvements: * Replace Curses mvprintw() with va_list-based equivalent. (#1509. Thanks to John Marshall and Andreas Tille) * Fixed some clang-13 warning messages. (#1506) * Improve quoting of options in "samtools import" tests. (#1466. Thanks to John Marshall) * Fixed a faulty test which caused test harness failures on NetBSD. (#1520) ------------------------------------------------------------------------------ bcftools - changes v1.14 ------------------------------------------------------------------------------ Changes affecting the whole of bcftools, or multiple commands: * New `--regions-overlap` and `--targets-overlap` options which address a long-standing design problem with subsetting VCF files by region. BCFtools recognize two sets of options, one for streaming (`-t/-T`) and one for index-gumping (`-r/-R`). They behave differently, the first includes only records with POS coordinate within the regions, the other includes overlapping regions. The two new options allow to modify the default behaviour, see the man page for more details. * The `--output-type` option can be used to override the default compression level Changes affecting specific commands: * bcftools annotate - when `--set-id` and `--remove` are combined, `--set-id` cannot use tags deleted by `--remove`. This is now detected and the program exists with an informative error message instead of segfaulting (#1540) - while non-symbolic variation are uniquely identified by POS,REF,ALT, symbolic alleles starting at the same position were indistinguishable. This prevented correct matching of records with the same positions and variant type but different length given by INFO/END (samtools/htslib@60977f2). When annotating from a VCF/BCF, the matching is done automatically. When annotating from a tab-delimited text file, this feature can be invoked by using `-c INFO/END`. - add a new '.' modifier to control whether missing values should be carried over from a tab-delimited file or not. For example: -c TAG .. adds TAG if the source value is not missing. If TAG exists in the target file, it will be overwritten -c .TAG .. adds TAG even if the source value is missing. This can overwrite non-missing values with a missing value and can create empty VCF fields (`TAG=.`) * bcftools +check-ploidy - by default missing genotypes are not used when determining ploidy. With the new option `-m, --use-missing` it is possible to use the information carried in the missing and half-missing genotypes (e.g. ".", "./." or "./1") * bcftools concat - new `--ligate-force` and `--ligate-warn` options for finer control of `-l, --ligate` behaviour in imperfect overlaps. The new default is to throw an error when sites present in one chunk but absent in the other are encountered. To drop such sites and proceed, use the new `--ligate-warn` option (previously this was the default). To keep such sites, use the new `--ligate-force` option (#1567). * bcftools consensus: - Apply mask even when the VCF has no notion about the chromosome. It was possible to encounter this problem when `contig` lines were not present in the VCF header and no variants were called on that chromosome (#1592) * bcftools +contrast: - support for chunking within map/reduce framework allowing to collect NASSOC counts even for empty case/control sample sets (#1566) * bcftools csq: - bug fix, compound indels were not recognised in some cases (#1536) - compound variants were incorrectly marked as 'inframe' even when stop codon would occur before the frame was restored (#1551) - bug fix, FORMAT/BCSQ bitmasks could have been assigned incorrectly to some samples at multiallelic sites, a superset of the correct consequences would have been set (#1539) - bug fix, the upstream stop could be falsely assigned to all samples in a multi-sample VCF even if the stop was relevant for a single sample only (#1578) - further improve the detection of mismatching chromosome naming (e.g. "chrX" vs "X") in the GFF, VCF and fasta files * bcftools merge: - keep (sum) INFO/AN,AC values when merging VCFs with no samples (#1394) * bcftools mpileup: - new --indel-size option which allows to increase the maximum considered indel size considered, large deletions in long read data are otherwise lost. * bcftools norm: - atomization now supports Number=A,R string annotations (#1503) - assign as many alternate alleles to genotypes at multiallelic sites in the`-m +` mode, disregarding the phase. Previously the program assumed to be executed as an inverse operation of `-m -`, but when that was not the case, reference alleles would have been filled instead of multiple alternate alleles (#1542) * bcftools sort: - increase accuracy of the --max-mem option limit, previously the limit could be exceeded by more than 20% (#1576) * bcftools +trio-dnm: - new `--with-pAD` option to allow processing of VCFs without FORMAT/QS. The existing `--ppl` option was changed to the analogous `--with-pPL` * bcftools view: - the functionality of the option --compression-level lost in 1.12 has been restored -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. |