Thread: [Samtools-help] Picard Release 1.35 (Page 2)

Brought to you by: awhitwham, bhandsaker, daviesrob, jenniferliddle, and 5 others

samtools-help

[Samtools-help] Picard Release 1.35

From: Alec W. <al...@br...> - 2010-11-15 17:01:06

Picard release 1.35
15 November 2010

Nothing terribly exciting in this release...

- CalculateHsMetrics.java: Added validation that bait and target files are the same reference as the reads.

- ExtractSequences.java: Added a new tool to extract subsequences from a fasta file.

- Provide more informative error message when countMismatches has an ArrayIndexOutOfBoundsException.


-Alec

Re: [Samtools-help] [Samtools-devel] Picard Release 1.35

From: Alec W. <al...@br...> - 2010-11-30 18:32:41

Hi Hyun,

[CCing samtools-help because we try to share the knowledge of the 
community...]

SamToFastq is not designed to be very efficient with memory.  It could 
be implemented better, but it hasn't been a focus for us.  The problem 
is that is writes out the two FASTQs in the same order, and if one end 
of the pair is not near the other in the input BAM, then the first end 
encountered is saved in memory until the second is found.  If there are 
lots of pairs with one end unmapped and without a coordinate, and/or 
chimeric reads, or long jumping reads, then this map can cause the 
program to run out of memory.

The solution is to feed it a queryname-sorted file so that the two mates 
are next to each other in the input BAM.  You could sort the file (with 
samtools or SortSam) and save it, and then feed the queryname sorted 
file to SamToFastq.  Alternately, you could have the sort program write 
to stdout, and then have SamToFastq read the sorted file from stdin.  
Details on stdin and stdout for Picard can be found here: 
http://sourceforge.net/apps/mediawiki/picard/index.php?title=Main_Page#Q:_Can_Picard_programs_read_from_stdin_and_write_to_stdout.3F

-Alec

On 11/30/10 1:05 PM, Hyun Min Kang wrote:
> Hi Alec,
>
> We are using picard 1.35's SamtoFastq.jar to convert a 20GB Exome BAM file to FASTQ file, and encountered the following errors. Even if we increased the memory to 16GB, it didn't resolve the errors. Do you have an idea why these error happen? The BAM files contains quite a large number of QC failed reads and duplicates.
>
> Thanks,
> Hyun.
>
> Error message is:
> Runtime.totalMemory()=9090039808
> Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
>          at java.lang.String.<init>(String.java:347)
>          at net.sf.samtools.util.StringUtil.bytesToString(StringUtil.java:259)
>          at net.sf.samtools.BAMRecord.decodeReadName(BAMRecord.java:325)
>          at net.sf.samtools.BAMRecord.getReadName(BAMRecord.java:214)
>          at net.sf.picard.sam.SamToFastq.doPaired(SamToFastq.java:148)
>          at net.sf.picard.sam.SamToFastq.doWork(SamToFastq.java:111)
>          at net.sf.picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:156)
>          at net.sf.picard.sam.SamToFastq.main(SamToFastq.java:96)
>
> samtools flagstat results are :
> 199846817 in total
> 25275848 QC failure
> 61789693 duplicates
> 170451665 mapped (85.29%)
> 131246536 paired in sequencing
> 65623268 read1
> 65623268 read2
> 113393300 properly paired (86.40%)
> 113532490 with itself and mate mapped
> 302704 singletons (0.23%)
> 100912 with mate mapped to a different chr
> 77082 with mate mapped to a different chr (mapQ>=5)
>
>
> On Nov 15, 2010, at 12:00 PM, Alec Wysoker wrote:
>
>    
>> Picard release 1.35
>> 15 November 2010
>>
>> Nothing terribly exciting in this release...
>>
>> - CalculateHsMetrics.java: Added validation that bait and target files are the same reference as the reads.
>>
>> - ExtractSequences.java: Added a new tool to extract subsequences from a fasta file.
>>
>> - Provide more informative error message when countMismatches has an ArrayIndexOutOfBoundsException.
>>
>>
>> -Alec
>> ------------------------------------------------------------------------------
>> Centralized Desktop Delivery: Dell and VMware Reference Architecture
>> Simplifying enterprise desktop deployment and management using
>> Dell EqualLogic storage and VMware View: A highly scalable, end-to-end
>> client virtualization framework. Read more!
>> http://p.sf.net/sfu/dell-eql-dev2dev_______________________________________________
>> Samtools-devel mailing list
>> Sam...@li...
>> https://lists.sourceforge.net/lists/listinfo/samtools-devel
>>      
>

[Samtools-help] Picard Release 1.36

From: Alec W. <al...@br...> - 2010-12-06 15:32:37

Picard Release 1.36 6 December 2010

- BAMIndexMetaData.java: Make public getAlignedRecordCount, 
getUnalignedRecordCount, getNoCoordinateCount. Improve interface to bam 
index metaData - public getMetaData in AbstractBamFileIndex.

- Support bgzipped SAM files in SAMFileReader.

- AbstractAlignmentMerger.java: Allow Sam or Bam output.

- FixMateInformation.java; When merging the input files, if they are all 
sorted then take advantage of that to produce a sorted input iterator.

- Soft-clip CIGAR operator inside of hard-clip operator is now 
considered valid.

- Non-standard SAM header tag values are no longer typed. They are all 
stored as strings. Any existing files that have a non-standard header 
tag value with a value type encoded in it will need to be repaired. 
These header tags will parse, but may not yield the expected results. 
AbstractSAMHeaderRecord.setAttribute(String, Object) has been 
deprecated. Use AbstractSAMHeaderRecord(String, String) instead. 
AbstractSAMHeaderRecord.getAttribute and .getAttributes have been 
changed to return String and Map<String, String>, respectively. Existing 
code that called these methods may need to be changed, in the cases 
where other than a String return value was expected.

-Alec

[Samtools-help] Picard Release 1.37

From: Alec W. <al...@br...> - 2010-12-20 14:24:44

Picard Release 1.37
20 December 2010

- SamRecordIntervalIteratorFactory.java: When not using index, halt 
iteration as soon as a read is seen that is after the last interval 
requested to avoid uselessly reading the entire file.

- Changes to IntervalList to ensure that only intervals defined against 
valid sequences can be loaded from file.

- MetricsFile.read will not throw an exception if the EOF is reached in 
before any metrics are read.  Instead an empty metrics file object is 
returned

- CollectAlignmentSummaryMetrics.java: If no reads are found than an 
empty unpaired category metric is output

- MarkDuplicates.java: The SAM Header is now parsed for libaries in 
order to try to pre-populate the metrics objects because otherwise there 
would be no duplication metrics objects for a library if no reads were found

- build.xml: Fix dependencies for package-commands target.

- Change behavior of RevertSam so that if 
REMOVE_DUPLICATE_INFORMATION=false, then the output can have records 
with the duplicate flag set despite being unaligned.

- CollectAlignmentSummaryMetrics.java: Reads shorter than 
ADAPTER_MATCH_LENGTH are not considered adapters.
Fixes ArrayIndexOutOfBoundsException from samtools-dev when 0-length read

-Alec

Re: [Samtools-help] Picard Release 1.37

From: Alec W. <al...@br...> - 2011-01-03 18:33:05

Picard release 1.38
3 January 2011

- Histogram.java: More efficient implementation of 
Historam.getStandardDeviation() and unit tests for the same.

- First version of a multi-metrics tool. Still fairly primitive.

- Added IntervalTreeMap.

- CommandLineProgram.java: Changed tmpdir.mkdir() to tmpdir.mkdirs()

- Suppress validation for zero-length read bases if SAMRecord is not 
primary.

-Alec

[Samtools-help] Picard release 1.39 delayed due to SourceForge outage

From: Alec W. <al...@br...> - 2011-01-31 15:30:28

Hi Folks,

SourceForge was attacked last week, and as a result has temporarily shut 
down various services, including file upload.  Details here: 
http://sourceforge.net/blog/sourceforge-attack-full-report/

Picard release 1.39 is delayed until SourceForge services are restored.

-Alec

[Samtools-help] Picard release 1.39

From: Alec W. <al...@br...> - 2011-02-02 14:09:46

Picard release 1.39
2 February 2011

- FastqToSam.java: Added options to set the sequencing center, PI, DS, 
DT tags in read group.

- CollectGcBiasMetrics.java: Fixed a bug whereby windows from a 
reference sequence were ignored if the reference sequence had no reads 
aligned to it.

- CollectGcBiasMetrics; When counting errors in GC window, add # of 
inserted bases and # of deleted bases to # of mismatched bases.

- CommandLineParser.java: Eliminate requirement that fields annotated 
with @Option and @Usage be public.



SourceForge appears to be partially functional now.  I am not able to 
browse the SVN repository via HTTP, but other SVN operations seem to be 
working.

-Alec

[Samtools-help] Picard release 1.40

From: Alec W. <al...@br...> - 2011-02-15 15:46:42

Picard release 1.40
15 February 2011

- CollectMultipleMetrics.java: Made sure that the list of programs to 
run is unique.

- Faster implementation of MeanQualityByCycle that cuts off about 2/3 of 
the runtime.

- Adds a .equals method to MetricBase that compares the values for all 
public fields in the class; modifies MetricFile to have separate 
comparison methods for header, metrics, and histogram and to call all 
three of them in the .equals method.  FormatUtil now handles Bytes.

- QualityScoreDistribution.java, MetricsFile.java: Fixed to cope with 
the situation when there are no valid bases in an input sam/bam file.

- Hack to fix strange problem with reading BAM via HTTP from SourceForge.


-Alec

[Samtools-help] Picard release 1.42

From: Alec W. <al...@br...> - 2011-03-29 14:18:44

Picard release 1.42
29 March 2011

- SAMRecordQueryNameComparator.java: Augment comparator to test 
not-primary flag and hit-index (HI) tag value, if present. Encapsulate 
read name comparator in case it gets changed.

- Restore value of SAMFileReader.defaultValidationStringency in various 
places where it is changed, so that methods that are typically main 
programs can be called in the context of some other program, and don't 
mess up the original validation stringency.

- SeekableHTTPStream.java: Eliminate work-around for SourceForge HTTP 
problem http://sourceforge.net/apps/trac/sourceforge/ticket/18367, since 
it no longer seems to be a problem.

- Added a little progress logging to SortSam.

- Changed SortingCollection to use a TreeSet in place of a 
PriorityQueue; this results in a speed up on the order of ~20% for the 
merging phase of SortingCollection when there are large numbers of temp 
files in use.

- CommandLineProgram.java: Modified to throw the original error, even if 
there is an exception when writing out the final log lines.

- SortingCollection.java: Make SortingCollection in-memory iteration 
destructive by default in order to reduce memory footprint.  If you want 
to call iterator() on a SortingCollection more than once, call 
setDestructiveIteration(false) to ensure that this will be allowed.  For 
SortingCollections that are large enough to require spilling to disk, 
iteration is non-destructive currently.

- Changes to centralize the size of buffers use in various buffering 
file read/writing classes and default them to 128kb.

- Add option to BAM index reading code to disable memory mapping of 
index file.  Memory-mapping can be problematic in Java because there is 
no way to force the release of a memory map before the garbage collector 
decides to do so, and therefore if many indices are opened and then 
closed, the JVM can run out of space in which to memory map.  By calling 
SAMFileReader.enableIndexMemoryMapping(false), RandomAccessFile is used 
rather than memory mapping, which results in slower performance but 
timely release of resources.  Change courtesy of Bob Handsaker.

-Alec

[Samtools-help] how to analyze vcf version 4.1 generated by mpileup and bcftools

From: Cliff Y. <mm...@ho...> - 2011-04-07 20:36:28

Hi,

I used samtools mpileup and bcftools to get variant vcf files and noticed the vcf format is 4.1. I then tried vcftools to analyze the data but got the following  error message:


VCFtools - v0.1.3
(C) Adam Auton 2009

Error:VCF version must be v4.0:
You are using version VCFv4.1

Is there any tool that I can use for vcf4.1?

Thanks

-C

Re: [Samtools-help] how to analyze vcf version 4.1 generated by mpileup and bcftools

From: Heng Li <lh...@sa...> - 2011-04-07 20:55:46

Samtools takes the GL/PL ordering required by the upcoming VCF-4.1. Thus it labels 4.1 in the header. However, the release of VCF-4.1 is delayed, which is causing the inconsistency. For now, you may just replace "VCFv4.1" with "VCFv4.0". This will have no side effect for both samtools and the current version of vcftools.

Heng

On Apr 7, 2011, at 4:36 PM, Cliff Yiu wrote:

> Hi,
> 
> I used samtools mpileup and bcftools to get variant vcf files and noticed the vcf format is 4.1. I then tried vcftools to analyze the data but got the following  error message:
> 
> 
> VCFtools - v0.1.3
> (C) Adam Auton 2009
> 
> Error:VCF version must be v4.0:
> You are using version VCFv4.1
> 
> Is there any tool that I can use for vcf4.1?
> 
> Thanks
> 
> -C
> 
> 
> ------------------------------------------------------------------------------
> Xperia(TM) PLAY
> It's a major breakthrough. An authentic gaming
> smartphone on the nation's most reliable network.
> And it wants your games.
> http://p.sf.net/sfu/verizon-sfdev_______________________________________________
> Samtools-help mailing list
> Sam...@li...
> https://lists.sourceforge.net/lists/listinfo/samtools-help



-- 
 The Wellcome Trust Sanger Institute is operated by Genome Research 
 Limited, a charity registered in England with number 1021457 and a 
 company registered in England with number 2742969, whose registered 
 office is 215 Euston Road, London, NW1 2BE.

Re: [Samtools-help] how to analyze vcf version 4.1 generated by mpileup and bcftools

From: Petr D. <pd...@sa...> - 2011-04-08 18:25:52

Hi,

the perl script and API from the VCFtools package support VCFv4.1

Best,
Petr


On Thu, 2011-04-07 at 15:36 -0500, Cliff Yiu wrote:
> Hi,
> 
> I used samtools mpileup and bcftools to get variant vcf files and
> noticed the vcf format is 4.1. I then tried vcftools to analyze the
> data but got the following  error message:
> 
> 
> VCFtools - v0.1.3
> (C) Adam Auton 2009
> 
> Error:VCF version must be v4.0:
> You are using version VCFv4.1
> 
> Is there any tool that I can use for vcf4.1?
> 
> Thanks
> 
> -C
> 
> 
> ------------------------------------------------------------------------------
> Xperia(TM) PLAY
> It's a major breakthrough. An authentic gaming
> smartphone on the nation's most reliable network.
> And it wants your games.
> http://p.sf.net/sfu/verizon-sfdev
> _______________________________________________ Samtools-help mailing list Sam...@li... https://lists.sourceforge.net/lists/listinfo/samtools-help




-- 
 The Wellcome Trust Sanger Institute is operated by Genome Research 
 Limited, a charity registered in England with number 1021457 and a 
 company registered in England with number 2742969, whose registered 
 office is 215 Euston Road, London, NW1 2BE.

[Samtools-help] Picard release 1.43

From: Alec W. <al...@br...> - 2011-04-11 14:23:14

Picard release 1.43
12 April 2011

- Enable MergeBamAlignment to handle multiple alignments for the same 
read.  There must be only one primary alignment, and alignments for mate 
pairs must be correlated with HI (hit index) tag.

-Alec

[Samtools-help] Picard release 1.45

From: Alec W. <al...@br...> - 2011-05-09 14:27:08

Picard release 1.45
9 May 2011

- changes to avoid NPE when calling equals() method on SAMRecord

- New program CollectCDnaMetrics -- beta release

- Simplified CollectAlignmentSummaryMetrics.java and added 
PF_ALIGNED_BASES, and PF_MISMATCH_RATE to AlignedSummaryMetrics, 
CollectAlignmentSummaryMetrics.

- BAMRemoteFileTest.java: Move location of HTTP-accessed BAM from 
SourceForge web to Broad web, because SF doesn't serve this file 
reliably (perhaps because BAM is GZIP-encoded?)

- CollectAlignmentSummaryMetrics.java, MeanQualityByCycle.java, 
QualityScoreDistribution.java: Ignore non-primary alignments when 
gathering metrics.

- Refactored AbstractTextFileParser, TabbedTextFileParser, and 
BasicTextFileParser as AbstractInputParser, TabbedInputParser, and 
BasicInputParser.  All now have additional constructors that take 
InputStreams, and the old constructors that take Files are still there.  
Except for the name change, they should continue to work the same as before.

- ExtractIlluminaBarcodes.java: Fixed Option for BASECALLS_DIR

- CollectAlignmentSummaryMetrics.java: Fixed PF_HQ_ERROR_RATE, overflow 
error due to previous simplification of CollectAlignmentSummaryMetrics.java

- RevertSam.java: Do not copy non-primary alignments into reverted SAM.

- CommandLineProgram.java: Added an output of the elapsed time when a 
program ends.

- Changes to ensure that cleanup() is called on all instances of 
SortingCollection and SortingLongCollection after use, to ensure that 
temporary files are cleaned up ASAP.

- SeekableHTTPStream.java: Allow FTP as well as HTTP URL connection 
(patch courtesy of Lance Frohman).

-Alec

[Samtools-help] Picard release 1.46

From: Alec W. <al...@br...> - 2011-05-23 15:15:52

Picard release 1.46
23 May 2011

- SeekableHTTPStream.java: Revert change to support FTP URLs because 
they aren't seekable.

- New program CollectRnaSeqMetrics.java.

- SAMRecord.java: Added SAMRecord.getReferencePositionAtReadPosition.

- TabbedTextFileWithHeaderParser.java: added ability to retrieve full 
(non-split) line from Row.

- Added option of providing a header array to 
TabbedTextFileWithHeaderParser.

- Fix bug in BasicInputParser where getCurrentLine and 
getCurrentLineNumber methods return information about  the *next* line 
rather than the current one.

- AbstractInputParser.java: Printing out offending line when parser fails.

- HsMetricsCalculator.java: Fix for a bug in calculating the 
HS_PENALTY_nnX numbers that was triggered if the input sequence data 
provided massively more coverage than needed to cover 80% of targets at 
nX.  In cases where the coverage was below, at or modestly above target 
coverage levels this wasn't a problem.

- FixMateInformation.java: Fix to correctly rename bam index file along 
without bam output when creating index.

- Changes to CollectGcBias metrics to remove a number of poorly tested 
and documented summary metrics and replace them with two new ones called 
AT_DROPOUT and GC_DROPOUT that mirror the metrics produced by Illumina.

- Allow for other AdapterPairs in ClippingUtility than just the ones 
defined in IlluminaUtil.

-Alec