#127 Parallel gzip - pigz

[Matthew King]

Would it be possible to allow Celera to select which compression program to use? There is a parallel implementation of gzip called pigz that can run very fast and compresses it to the exact same format. By default it runs on 8 cores.
http://www.zlib.net/pigz/

[Brian Walenz]

I like the idea, but I doubt it'll help much. pigz only parallelizes the compression. The assembler only compresses overlap output, and that compute is already parallel and (should be) consuming all CPU. pigz doesn't parallelize decompression, which is where we could use it most.

There is only one place that would benefit from pigz -- AS_OVS/AS_OVS_overlapFile.c -- if you want to give it a shot (replace 'gzip' with 'pigz', recompile).

We probably won't get to implementing this until late January.

[Alexie]

another option is pbzip2 with -1 (fast). Might be useful as it allows for parallel decompression (if the bzip2 file has been compressed in parallel).

[Jason Miller]

Thank you for this excellent suggestion. It certainly would be possible since we already support three options (none, gzip, bzip2). Let's consider feature request this for an upcoming release.