Re: [Samtools-help] picard illuminabasecallstofastq

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 454-5900

Hi Alec,

Thanks for the suggestions. First items is likely irrelevant since the same issue was present before the recent 1.107 update with the BclReader buffer size. For the second and third item, are they related. If I ask for 4  NUM_PROCESSORS, will each process use MAX_READS_IN_RAM_PER_TILE so that the memory footprint will quadruple. If that is the case than it makes more sense so adjust MAX_READS_IN_RAM_PERTILE and NUM_PROC with java –Xmx.

Otherwise, it feels as if things are being kept in memory longer than they need to be. I thought what the tool would do is to read illumina files and write fastq files, therefore no memory needs to be utilized at all. Maybe I do not have a clear understanding of the inner workings of the tools.

Thanks for the suggestions. I will play around with those. What is the general consensus for running this particular tool. Any recommendations,

From: Alec Wysoker <al...@br...<mailto:al...@br...>>
Date: Wednesday, January 29, 2014 3:55 PM
To: "Abaan, Ogan (NIH/NCI) [E]" <ab...@ma...<mailto:ab...@ma...>>
Cc: "sam...@li...<mailto:sam...@li...>" <sam...@li...<mailto:sam...@li...>>
Subject: Re: [Samtools-help] picard illuminabasecallstofastq

Hi Ogan,

There are a few things you can do to reduce the memory footprint:

  *   A change was made recently whicih caused the buffer size for BclReader to be larger.  To restore to the previous size, use -Dsamjdk.buffer_size=8192
  *   You can reduce the value for MAX_READS_IN_RAM_PER_TILE
  *   You can reduce NUM_PROCESSORS

-Alec

On Jan 29, 2014, at 3:30 PM, Ogan <ab...@gm...<mailto:ab...@gm...>> wrote:

sorry for ignoring that, here is the commandline and stack trace from a failed run.

best

Ogan

net.sf.picard.illumina.IlluminaBasecallsToFastq BASECALLS_DIR=/home/solexa/131113_SN7001190R_0118_BC2MPCACXX_COMBINED/Data/Intensities/BaseCalls LANE=6 OUTPUT_PREFIX=6_131113_SN7001190R_0118_BC2MPCACXX_COMBINED RUN_BARCODE=6_131113_SN7001190R_0118_BC2MPCACXX_COMBINED READ_STRUCTURE=101T8B101T NUM_PROCESSORS=4 FORCE_GC=false READ_NAME_FORMAT=ILLUMINA TMP_DIR=[/home/solexa/Ogan/alignments/BC2MPCACXX/tmpdir.8bDUp8PC] VERBOSITY=ERROR    ADAPTERS_TO_CHECK=[INDEXED, DUAL_INDEXED, NEXTERA_V2, FLUIDIGM] APPLY_EAMSS_FILTER=true MAX_READS_IN_RAM_PER_TILE=1200000 MINIMUM_QUALITY=2 QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false
OpenJDK 64-Bit Server VM warning: You have loaded library /home/abaano/bin/PICARD/picard-tools-1.107/libIntelDeflater.so which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
[Wed Jan 29 11:52:30 EST 2014] Executing as abaano@Mueller on Linux 3.5.4 amd64; OpenJDK 64-Bit Server VM 1.7.0_45-b31; Picard version: 1.107(1667) IntelDeflater
ERROR    Exception in thread "pool-1-thread-3" java.lang.OutOfMemoryError: Java heap space
    at java.io.BufferedInputStream.<init>(BufferedInputStream.java:195)
    at net.sf.picard.illumina.parser.readers.BclReader.<init>(BclReader.java:107)
    at net.sf.picard.illumina.parser.BclParser$1.<init>(BclParser.java:79)
    at net.sf.picard.illumina.parser.BclParser.makeCycleFileParser(BclParser.java:77)
    at net.sf.picard.illumina.parser.PerTilePerCycleParser.seekToTile(PerTilePerCycleParser.java:121)
    at net.sf.picard.illumina.parser.PerTilePerCycleParser.initialize(PerTilePerCycleParser.java:76)
    at net.sf.picard.illumina.parser.BclParser.<init>(BclParser.java:60)
    at net.sf.picard.illumina.parser.IlluminaDataProviderFactory.makeParser(IlluminaDataProviderFactory.java:313)
    at net.sf.picard.illumina.parser.IlluminaDataProviderFactory.makeDataProvider(IlluminaDataProviderFactory.java:203)
    at net.sf.picard.illumina.IlluminaBasecallsConverter$TileReader.process(IlluminaBasecallsConverter.java:468)
    at net.sf.picard.illumina.IlluminaBasecallsConverter$TileReadAggregator$2.run(IlluminaBasecallsConverter.java:569)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
2014-01-29 14:09:17    IlluminaBasecallsConverter    Failure encountered in worker thread; attempting to shut down remaining worker threads and terminate ...
java.lang.InterruptedException
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:503)
    at net.sf.picard.illumina.IlluminaBasecallsConverter$TileReadAggregator.awaitWorkComplete(IlluminaBasecallsConverter.java:620)
    at net.sf.picard.illumina.IlluminaBasecallsConverter.doTileProcessing(IlluminaBasecallsConverter.java:243)
    at net.sf.picard.illumina.IlluminaBasecallsToFastq.doWork(IlluminaBasecallsToFastq.java:154)
    at net.sf.picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:179)
    at net.sf.picard.cmdline.CommandLineProgram.instanceMainWithExit(CommandLineProgram.java:120)
    at net.sf.picard.illumina.IlluminaBasecallsToFastq.main(IlluminaBasecallsToFastq.java:301)
Exception in thread "pool-1-thread-1" net.sf.picard.PicardException: Error in writing fastq file /home/solexa/Ogan/alignments/BC2MPCACXX/6_131113_SN7001190R_0118_BC2MPCACXX_COMBINED.1.fastq
    at net.sf.picard.fastq.BasicFastqWriter.write(BasicFastqWriter.java:66)
    at net.sf.picard.illumina.IlluminaBasecallsToFastq$FastqRecordsWriter.write(IlluminaBasecallsToFastq.java:328)
    at net.sf.picard.illumina.IlluminaBasecallsToFastq$FastqRecordsWriter.write(IlluminaBasecallsToFastq.java:322)
    at net.sf.picard.illumina.IlluminaBasecallsToFastq$FastqRecordsWriter.write(IlluminaBasecallsToFastq.java:307)
    at net.sf.picard.illumina.IlluminaBasecallsConverter$TileReadAggregator$3.run(IlluminaBasecallsConverter.java:742)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
[Wed Jan 29 14:09:47 EST 2014] net.sf.picard.illumina.IlluminaBasecallsToFastq done. Elapsed time: 137.28 minutes.
Runtime.totalMemory()=30860115968
To get help, see http://picard.sourceforge.net/index.shtml#GettingHelp
Exception in thread "main" net.sf.picard.PicardException: Failure encountered in worker thread; see log for details.
    at net.sf.picard.illumina.IlluminaBasecallsConverter.doTileProcessing(IlluminaBasecallsConverter.java:246)
    at net.sf.picard.illumina.IlluminaBasecallsToFastq.doWork(IlluminaBasecallsToFastq.java:154)
    at net.sf.picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:179)
    at net.sf.picard.cmdline.CommandLineProgram.instanceMainWithExit(CommandLineProgram.java:120)
    at net.sf.picard.illumina.IlluminaBasecallsToFastq.main(IlluminaBasecallsToFastq.java:301)

On 1/29/14 2:57 PM, Alec Wysoker wrote:
Hi Abaano,

Please supply as much of the information describe here as you can (obviously not stack trace since program didn't crash): http://picard.sourceforge.net/index.shtml#GettingHelp

Thanks, Alec

On Jan 29, 2014, at 2:46 PM, Ogan <ab...@gm...<mailto:ab...@gm...>> wrote:

Dear all,

I am using picards illuminabasecallstofastq to convert hiSeq lane
basecalls to fq files for alignments. I observe that the whole process
uses a lot of memory, more than 32G (so far it has been ok with 48G) for
one lane of HiSeq data. I was curious if that is normal behavior, and if
it is, what is being kept in memory. I have seen this since v1.1nn
including 1.107 with GC cleanup on.

Thank you

------------------------------------------------------------------------------
WatchGuard Dimension instantly turns raw network data into actionable
security intelligence. It gives you real-time visual feedback on key
security issues and trends.  Skip the complicated setup - simply import
a virtual appliance and go from zero to informed in seconds.
http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk
_______________________________________________
Samtools-help mailing list
Sam...@li...<mailto:Sam...@li...>
https://lists.sourceforge.net/lists/listinfo/samtools-help