#127 Parallel gzip - pigz

open
Feature (48)
5
2011-04-04
2010-12-15
No

Would it be possible to allow Celera to select which compression program to use? There is a parallel implementation of gzip called pigz that can run very fast and compresses it to the exact same format. By default it runs on 8 cores.
http://www.zlib.net/pigz/

Discussion

  • Brian Walenz

    Brian Walenz - 2010-12-15

    I like the idea, but I doubt it'll help much. pigz only parallelizes the compression. The assembler only compresses overlap output, and that compute is already parallel and (should be) consuming all CPU. pigz doesn't parallelize decompression, which is where we could use it most.

    There is only one place that would benefit from pigz -- AS_OVS/AS_OVS_overlapFile.c -- if you want to give it a shot (replace 'gzip' with 'pigz', recompile).

    We probably won't get to implementing this until late January.

     
  • Alexie

    Alexie - 2011-03-30

    another option is pbzip2 with -1 (fast). Might be useful as it allows for parallel decompression (if the bzip2 file has been compressed in parallel).

     
  • Jason Miller

    Jason Miller - 2011-04-04

    Thank you for this excellent suggestion. It certainly would be possible since we already support three options (none, gzip, bzip2). Let's consider feature request this for an upcoming release.

     
  • Jason Miller

    Jason Miller - 2011-04-04
    • assigned_to: nobody --> brianwalenz
     

Log in to post a comment.

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:

JavaScript is required for this form.





No, thanks