samtools-announce — announcement of releases and critical bug fixes (low trafic)

[Samtools-announce] Picard 1.12 released

[Alec W. <al...@br...>]

Release notes for Picard 1.12
Mon Jan 4 2010

- Simplified SAMSequenceRecord.isSameSequence to compare only 
sequenceIndex, sequenceLength, and either MD5 or name, so that 
variations in sequence name are tolerated.

- All integer tags are now loaded from BAM as type Integer, regardless 
of whether they are stored at Byte, Char, Short or Integer.  They are 
now stored in BAM in the smallest encoding that will hold them.  
Unsigned integer is not allowed, although with validation stringency set 
to lenient or silent an unsigned integer can be read from a BAM.  For 
SAM, a value greater than a signed 32-bit int cannot be read.

- Add SortSam program.

- Add typed SAMRecord.getAttribute methods.

- Add SAMRecord.clearAttributes()

- Increase MAX_PHRED_SCORE to 93.

- Add overload of SequenceUtil.sumQualitiesOfMismatches that works with 
a subset of the reference instead of requiring the whole reference array.

- Add overload of SequenceUtil.calculateSamNmTag that doesn't require 
the entire reference sequence, but just the chunk overlapping the read.

- Added support for multiple histograms with the same key-space.

- Performance tweak: Change choice of when AsciiLineReaderImpl is used 
instead of BufferedLineReader, so it is used for all non-Linux systems.


-Alec

[Samtools-announce] Picard 1.11 releases

[Alec W. <al...@br...>]

Hi Folks,

Note that I skipped version 1.10 because the trailing zero caused 
problems for Sourceforge.

- Fix bug in which BAM linear index was being indexed improperly, 
leading to unnecessary SAMRecords being considered and then discarded in 
a query() operation.

- In CommandLineProgram, if customCommandLineValidation returns error 
messages, print usage after printing error message, as happens for 
standard command line validation
  errors.

- Add SAMFileWriter.getFileHeader() to public interface, and rename 
SAMFileWriterImpl.getHeader() to getFileHeader() for consistency.

- Eliminate instance-level log level in Log class, so that calling 
Log.setGlobalLogLevel affects all instances regardless of whether they 
were created before or after the call.

- Make Cigar code and decode public.

- Added Cigar.getReadLength().

- Fix possible bug in which BlockCompressedInputStream.checkTermination 
wasn't closing the file after checking, so if the code tried to delete 
the file before GC could close the file, deletion might fail.  Seemed to 
affect Windows only.

- After discovering that BufferedReader is faster than AsciiLineReader 
on Linux but slower on Mac, implemented class (AsciiLineReader) that 
detects OS and delegates to AsciiLineReaderImpl or BufferedLineReader as 
appropriate.  AsciiLineReaderImpl does byte=>char conversion itself 
rather than calling StringUtil.bytesToString() for better performance.  
Added close() to LineReader interface, and removed 
readLine(includeTerminator).

- In FastqToSam & SamToFastq, fix to not write out "null" in the quality 
header and to add an option to output only PF reads in the fastq files.

- Changed default behaviour of CreateSequenceDictionary to truncate 
sequence names at whitespace.


-Alec

[Samtools-announce] Picard 1.09 released

[Alec W. <al...@br...>]

Picard Release 1.09
Mon Dec 7 2009

Changes:

- Negative strand flag can be set on unmapped read

- Added Histogram.getSumOfValues() method to sum up the values in the 
bins without
  multiplying them by the bin's id value

- Added a new Histogram method for getting the cumulative propbability 
for a value when
  the histogram is being used to model a numerical distribution.

- Some significant changes to FastqToSam, mainly in what parameters are 
taken and
  required, to make the whole thing a bit more intuitive.

- Validation stringency is now respected in parsing of SAM file header.  
Note that because
  the header is parsed in SAMFileReader ctor, the only way to control 
the validation
  stringency used to parse the header is to call  
SAMFileReader.setDefaultValidationStringency.

- If validation stringency is not strict, validation errors are stored 
in the
  SAMFileHeader, so non-command-line clients (e.g. IGV) can access the 
errors
  programmatically.

- The text of the SAM header as it appeared in the input is a property 
on SAMFileHeader.
  Note that this is not a toString() operation.  Changes to the header 
after reading from
  the input are not reflected in this property.

- @CO comment lines are now handled, stored in SAMFileHeader, merged in 
MergeSamFiles, and
  written to output SAM file.

- After closing BGZF file being written, check for empty gzip block and 
throw exception if not found.

- Add call to FileDescriptor.sync() in BinaryCodec.close, if 
OutputStream is a
  FileOutputStream, to try as much  as possible to ensure that a BAM 
gets flushed to disk.

- Improve error reporting about bad sort order -- show the sort key that 
is appropriate for the sort order.


-Alec

[Samtools-announce] samtools-0.1.7 released

[Heng Li <lh...@sa...>]

Beta Release 0.1.7 (10 November, 2009)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Notable changes:

 * Improved the indel caller in complex scenariors, in particular for
   long reads. The indel caller is now able to make reasonable indel
   calls from Craig Venter capillary reads.

 * Rewrote single-end duplicate removal with improved
   performance. Paired-end reads are not touched.

 * Duplicate removal is now library aware. Samtools remove potential
   PCR/optical dupliates inside a library rather than across libraries.

 * SAM header is now fully parsed, although this functionality is not
   used in merging and so on.

 * In samtools merge, optionally take the input file name as RG-ID and
   attach the RG tag to each alignment.

 * Added FTP support in the RAZF library. RAZF-compressed reference
   sequence can be retrieved remotely.

 * Improved network support for Win32.

 * Samtools sort and merge are now stable.

Changes in other utilities:

 * Implemented sam2vcf.pl that converts the pileup format to the VCF
   format.

 * This release of samtools is known to work with the latest
   Bio-Samtools Perl module.

(0.1.7: 10 November 2009, r510)


-- 
 The Wellcome Trust Sanger Institute is operated by Genome Research 
 Limited, a charity registered in England with number 1021457 and a 
 company registered in England with number 2742969, whose registered 
 office is 215 Euston Road, London, NW1 2BE. 

[Samtools-announce] a new tool tabix is released

[Heng Li <lh...@sa...>]

Tabix is an indexer and data retriever for TAB-delimited genome position
files such as GFF/GTF, BED and even SAM. If you want to quickly retrieve
data in a specified region, you may like this small tool. Tabix also
comes with native HTTP/FTP support. Web developers may use this feature
to implement custom tracks or RESTful services.

Please go to the download page for source code. Detailed documentation
is on the samtools website.

best,

Heng


-- 
 The Wellcome Trust Sanger Institute is operated by Genome Research 
 Limited, a charity registered in England with number 1021457 and a 
 company registered in England with number 2742969, whose registered 
 office is 215 Euston Road, London, NW1 2BE. 

Re: [Samtools-announce] [Samtools-devel] Release samtools-0.1.6 & open tasks

[Lincoln S. <lin...@gm...>]

I have recently released Bio::DB::Sam version 1.04 to CPAN (the
Comprehensive Perl Archive Network). The only change was to update the
example bam file distributed with it so that all regression tests pass when
linked with samtools 1.06

Lincoln

On Thu, Sep 3, 2009 at 8:47 AM, Heng Li <lh...@sa...> wrote:

> Hello,
>
> Samtools-0.1.6 is released. The release note is appened in the end of
> this email. This release has a little compatibility issue with the Perl
> module Bio::DB::Sam due to a minor format change in BAM. The "./Build
> test" will fail, but to my knowledge, Bio::DB::Sam still works properly
> with samtools-0.1.6. If you have problems, please compile the perl
> module with samtools-0.1.5c instead.
>
> I have also added a page (http://samtools.sourceforge.net/opentask.shtml)
> describing a few open tasks which I do not have time to work on at the
> moment. If you are interested in these and happen to have time, you may
> help to improve samtools by contributing source codes. By the way, sorry
> to Bret Harry that I have not spared time to incorporate your patches
> about autoconf integration. I will do in future.
>
> Best,
>
> Heng
>
>
>
> Beta Release 0.1.6 (2 September, 2009)
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> Notable changes:
>
>  * In tview, do not show a blank screen when no reads mapped to the
>   corresponding region.
>
>  * Implemented native HTTP support in the BGZF library. Samtools is now
>   able to directly open a BAM file on HTTP. HTTP proxy is also
>   supported via the "http_proxy" environmental variable.
>
>  * Samtools is now compitable with the MinGW (win32) compiler and the
>   PDCurses library.
>
>  * The calmd (or fillmd) command now calculates the NM tag and replaces
>   MD tags if they are wrong.
>
>  * The view command now recognizes and optionally prints FLAG in HEXs or
>   strings to make a SAM file more friendly to human eyes. This is a
>   samtools-C extension, not implemented in Picard for the time
>   being. Please type `samtools view -?' for more information.
>
>  * BAM files now have an end-of-file (EOF) marker to facilitate
>   truncation detection. A warning will be given if an on-disk BAM file
>   does not have this marker. The warning will be seen on BAM files
>   generated by an older version of samtools. It does NO harm.
>
>  * New key bindings in tview: `r' to show read names and `s' to show
>   reference skip (N operation) as deletions.
>
>  * Fixed a bug in `samtools merge -n'.
>
>  * Samtools merge now optionally copies the header of a user specified
>   SAM file to the resultant BAM output.
>
>  * Samtools pileup/tview works with a CIGAR with the first or the last
>   operation is an indel.
>
>  * Fixed a bug in bam_aux_get().
>
>
> Changes in other utilies:
>
>  * Fixed wrong FLAG in maq2sam.
>
>
> (0.1.6: 2 September 2009, r453)
>
>
> --
>  The Wellcome Trust Sanger Institute is operated by Genome Research
>  Limited, a charity registered in England with number 1021457 and a
>  company registered in England with number 2742969, whose registered
>  office is 215 Euston Road, London, NW1 2BE.
>
>
> ------------------------------------------------------------------------------
> Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
> trial. Simplify your report design, integration and deployment - and focus
> on
> what you do best, core application coding. Discover what's new with
> Crystal Reports now.  http://p.sf.net/sfu/bobj-july
> _______________________________________________
> Samtools-devel mailing list
> Sam...@li...
> https://lists.sourceforge.net/lists/listinfo/samtools-devel
>



-- 
Lincoln D. Stein
Director, Informatics and Biocomputing Platform
Ontario Institute for Cancer Research
101 College St., Suite 800
Toronto, ON, Canada M5G0A3
416 673-8514
Assistant: Renata Musa <Ren...@oi...>

[Samtools-announce] Release samtools-0.1.6 & open tasks

[Heng Li <lh...@sa...>]

Hello,

Samtools-0.1.6 is released. The release note is appened in the end of
this email. This release has a little compatibility issue with the Perl
module Bio::DB::Sam due to a minor format change in BAM. The "./Build
test" will fail, but to my knowledge, Bio::DB::Sam still works properly
with samtools-0.1.6. If you have problems, please compile the perl
module with samtools-0.1.5c instead.

I have also added a page (http://samtools.sourceforge.net/opentask.shtml)
describing a few open tasks which I do not have time to work on at the
moment. If you are interested in these and happen to have time, you may
help to improve samtools by contributing source codes. By the way, sorry
to Bret Harry that I have not spared time to incorporate your patches
about autoconf integration. I will do in future.

Best,

Heng



Beta Release 0.1.6 (2 September, 2009)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Notable changes:

 * In tview, do not show a blank screen when no reads mapped to the
   corresponding region.

 * Implemented native HTTP support in the BGZF library. Samtools is now
   able to directly open a BAM file on HTTP. HTTP proxy is also
   supported via the "http_proxy" environmental variable.

 * Samtools is now compitable with the MinGW (win32) compiler and the
   PDCurses library.

 * The calmd (or fillmd) command now calculates the NM tag and replaces
   MD tags if they are wrong.

 * The view command now recognizes and optionally prints FLAG in HEXs or
   strings to make a SAM file more friendly to human eyes. This is a
   samtools-C extension, not implemented in Picard for the time
   being. Please type `samtools view -?' for more information.

 * BAM files now have an end-of-file (EOF) marker to facilitate
   truncation detection. A warning will be given if an on-disk BAM file
   does not have this marker. The warning will be seen on BAM files
   generated by an older version of samtools. It does NO harm.

 * New key bindings in tview: `r' to show read names and `s' to show
   reference skip (N operation) as deletions.

 * Fixed a bug in `samtools merge -n'.

 * Samtools merge now optionally copies the header of a user specified
   SAM file to the resultant BAM output.

 * Samtools pileup/tview works with a CIGAR with the first or the last
   operation is an indel.

 * Fixed a bug in bam_aux_get().


Changes in other utilies:

 * Fixed wrong FLAG in maq2sam.


(0.1.6: 2 September 2009, r453)


-- 
 The Wellcome Trust Sanger Institute is operated by Genome Research 
 Limited, a charity registered in England with number 1021457 and a 
 company registered in England with number 2742969, whose registered 
 office is 215 Euston Road, London, NW1 2BE. 

[Samtools-announce] samtools-0.1.5c released (new bug fixed)

[Heng Li <lh...@sa...>]

I am sorry for this additional email. Since the release of 0.1.5, my  
colleagues and I quickly found new bugs in it and so I replaced the  
packages at the download page but did not change the overall release  
number. The current version at the download page is "0.1.5c (r385)".  
Please make sure you are using this version. The changes are:

  * Made the source code compatible with Bio::DB::Sam (http://search.cpan.org/~lds/Bio-SamTools-1.00/lib/Bio/DB/Sam.pm 
)

  * Fixed a bug when piping is in use.

I am sorry for this change and hope the new version makes your work  
easier.

Heng

Begin forwarded message:

> From: Heng Li <lh...@sa...>
> Date: 7 July 2009 11:13:53 BST
> To: samtools help <sam...@li...>, sam...@li...
> Subject: samtools-0.1.5 released
>
> Beta Release 0.1.5 (7 July, 2009)
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> Notable changes:
>
> * Support opening a BAM alignment on FTP. Users can now use "tview" to
>   view alignments at the NCBI ftp site. Please read manual for more
>   information.
>
> * In library, propagate errors rather than exit or complain assertion
>   failure.
>
> * Simplified the building system and fixed compiling errors caused by
>   zlib<1.2.2.1.
>
> * Fixed an issue about lost header information when a SAM is imported
>   with "view -t".
>
> * Implemented "samtool.pl varFilter" which filters both SNPs and short
>   indels. This command replaces "indelFilter".
>
> * Implemented "samtools.pl pileup2fq" to generate FASTQ consensus from
>   pileup output.
>
> * In pileup, cap mapping quality at 60. This helps filtering when
>   different aligners are in use.
>
> * In pileup, allow to output variant sites only.
>
> * Made pileup generate correct calls in repetitive region. At the same
>   time, I am considering to implement a simplified model in SOAPsnp,
>   although this has not happened yet.
>
> * In view, added '-u' option to output BAM without compression. This
>   option is preferred when the output is piped to other commands.
>
> * In view, added '-l' and '-r' to get the alignments for one library  
> or
>   read group. The "@RG" header lines are now partially parsed.
>
> * Do not include command line utilities to libbam.a.
>
> * Fixed memory leaks in pileup and bam_view1().
>
> * Made faidx more tolerant to empty lines right before or after  
> FASTA >
>   lines.
>
>
> Changes in other utilities:
>
> * Updated novo2sam.pl by Colin Hercus, the key developer of novoalign.
>
>
> This release involves several modifications to the key code base which
> may potentially introduce new bugs even though we have tried to  
> minimize
> this by testing on several examples. Please let us know if you catch
> bugs.
>
> (0.1.5: 7 July 2009, r373)
>



-- 
 The Wellcome Trust Sanger Institute is operated by Genome Research 
 Limited, a charity registered in England with number 1021457 and a 
 company registered in England with number 2742969, whose registered 
 office is 215 Euston Road, London, NW1 2BE. 

[Samtools-announce] samtools-0.1.5 released

[Heng Li <lh...@sa...>]

Beta Release 0.1.5 (7 July, 2009)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Notable changes:

  * Support opening a BAM alignment on FTP. Users can now use "tview" to
    view alignments at the NCBI ftp site. Please read manual for more
    information.

  * In library, propagate errors rather than exit or complain assertion
    failure.

  * Simplified the building system and fixed compiling errors caused by
    zlib<1.2.2.1.

  * Fixed an issue about lost header information when a SAM is imported
    with "view -t".

  * Implemented "samtool.pl varFilter" which filters both SNPs and short
    indels. This command replaces "indelFilter".

  * Implemented "samtools.pl pileup2fq" to generate FASTQ consensus from
    pileup output.

  * In pileup, cap mapping quality at 60. This helps filtering when
    different aligners are in use.

  * In pileup, allow to output variant sites only.

  * Made pileup generate correct calls in repetitive region. At the same
    time, I am considering to implement a simplified model in SOAPsnp,
    although this has not happened yet.

  * In view, added '-u' option to output BAM without compression. This
    option is preferred when the output is piped to other commands.

  * In view, added '-l' and '-r' to get the alignments for one library  
or
    read group. The "@RG" header lines are now partially parsed.

  * Do not include command line utilities to libbam.a.

  * Fixed memory leaks in pileup and bam_view1().

  * Made faidx more tolerant to empty lines right before or after  
FASTA >
    lines.


Changes in other utilities:

  * Updated novo2sam.pl by Colin Hercus, the key developer of novoalign.


This release involves several modifications to the key code base which
may potentially introduce new bugs even though we have tried to minimize
this by testing on several examples. Please let us know if you catch
bugs.

(0.1.5: 7 July 2009, r373)



-- 
 The Wellcome Trust Sanger Institute is operated by Genome Research 
 Limited, a charity registered in England with number 1021457 and a 
 company registered in England with number 2742969, whose registered 
 office is 215 Euston Road, London, NW1 2BE. 

[Samtools-announce] SAMtools paper goes online (Open Access)

[Heng Li <lh...@sa...>]

Hello,

SAMtools paper has been accepted by Bioinformatics as an Applications  
Note. It is online now (Open Access):

http://bioinformatics.oxfordjournals.org/cgi/content/abstract/btp352

This paper gives a quick overview of the SAM format. It may be better  
than the specification if you do not want to dive into two many details.

I am sending to multiple mailing lists. I am sorry if you get this  
email multiple times.

Best,

Heng


-- 
 The Wellcome Trust Sanger Institute is operated by Genome Research 
 Limited, a charity registered in England with number 1021457 and a 
 company registered in England with number 2742969, whose registered 
 office is 215 Euston Road, London, NW1 2BE. 

[Samtools-announce] samtools-0.1.4 are released.

[Heng Li <lh...@sa...>]

Users are recommended to download the latest version because it has  
some bugs fixed.

Release notes:

Beta Release 0.1.4 (21 May, 2009)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Notable changes:

  * Added the 'rmdupse' command: removing duplicates for SE reads.

  * Fixed a critical bug in the indel caller: clipped alignments are not
    processed correctly.

  * Fixed a bug in the tview: gapped alignment may be incorrectly
    displayed.

  * Unified the interface to BAM and SAM I/O. This is done by
    implementing a wrapper on top of the old APIs and therefore old APIs
    are still valid. The new I/O APIs also recognize the @SQ header
    lines.

  * Generate the MD tag.

  * Generate "=" bases. However, the indel caller will not work when "="
    bases are present.

  * Enhanced support of color-read display (by Nils Homer).

  * Implemented the GNU building system. However, currently the building
    system does not generate libbam.a. We will improve this later. For
    the time being, `make -f Makefile.generic' is preferred.

  * Fixed a minor bug in pileup: the first read in a chromosome may be
    skipped.

  * Fixed bugs in bam_aux.c. These bugs do not affect other components  
as
    they were not used previously.

  * Output the 'SM' tag from maq2sam.

(0.1.4: 21 May 2009, r297)



-- 
 The Wellcome Trust Sanger Institute is operated by Genome Research 
 Limited, a charity registered in England with number 1021457 and a 
 company registered in England with number 2742969, whose registered 
 office is 215 Euston Road, London, NW1 2BE. 

[Samtools-announce] Release of samtools-0.1.3

[Heng Li <lh...@sa...>]

Hello,

I have just released samtools-0.1.3 (the C binding). The specification  
is also updated.

Heng


-- 
 The Wellcome Trust Sanger Institute is operated by Genome Research 
 Limited, a charity registered in England with number 1021457 and a 
 company registered in England with number 2742969, whose registered 
 office is 215 Euston Road, London, NW1 2BE.