[Algorithms] Comparing apples to apples : compression libraries

[John W. R. <jo...@si...>]

With pretty much every developer there comes a time when you need to use a
compression library; not because you want to 'zip' up a massive archive but
more because you want to compress some game data on the fly before you
transmit it over the network, or to shrink the size of a local cache or
something.  And, like pretty much every developer, you end up wandering
through a maze of open source libraries on the internet, each of which has
entirely different API, source code layout, and license agreement.  Just
figuring out to call 'compressData' (if you even can) is often hours of
wading through documentation and dealing with build/configuration issues.

 

For your convenience I am releasing a small code snippet that I wrote when I
was trying to evaluate a replacement compression library than the one we are
currently using.  The simple fact of the matter is that I don't believe any
other programmer should have to waste the same amount of stupid time I did
wrestling these libraries into a convenient form.

 

http://www.amillionpixels.us/test_compression_1.0.exe

 

 

Here is the complete API to my wrapper interface:

 

// Block compression/decompression library wrapper by John W. Ratcliff
http://www.codesuppository.blogspot.com/

 

enum CompressionType

{

  CT_INVALID,

  CT_CRYPTO_GZIP,       // The CryptoPP library implementation of GZIP
http://www.cryptopp.com

  CT_MINILZO,           // The MiniLZO library
http://www.oberhumer.com/opensource/lzo/

  CT_ZLIB,              // The ZLIB library   http://www.zlib.net/

  CT_BZIP,              // The BZIP library  http://www.bzip.org/

};

 

void *           compressData(const void *source,int len,int
&outlen,CompressionType type=CT_ZLIB);

void *           decompressData(const void *source,int clen,int &outlen);

void             deleteData(void* mem);

 

CompressionType  getCompressionType(const void *mem,int len);

const char      *getCompressionTypeString(CompressionType type);

 

};

 

This release supports four open source compression libraries.  The CRYPTOPP
implementation of GZIP, MINILZO, ZLIB, and BZIP.

 

This API simply supports block memory compression and decompression.  You
can compress a block of memory using any one of the four compressors.  The
compressed memory has a small header on it that indicates which compressor
was used, the size of the uncompressed memory block, and a CRC so that it
can easily and safely decompressed.

 

The test application, called 'test_compression', loads a roughly 10mb XML
file and runs it through each compressor and decompressor and measures the
performance characteristcs of each.  The results are as follows:

 

 

Testing Compression rate and speed with various compressors.

---------------------------------------------------------------

Compress:CT_CRYPTO  :FROM: 10,436,335 TO:  2,498,433   23%    1,170 MS

Compress:CT_MINILZO :FROM: 10,436,335 TO:  3,940,072   37%       97 MS

Compress:CT_ZLIB    :FROM: 10,436,335 TO:  3,299,771   31%      157 MS

Compress:CT_BZIP    :FROM: 10,436,335 TO:  2,270,695   21%    1,544 MS

---------------------------------------------------------------

Testing Decompression speed with various decompressors.

---------------------------------------------------------------

Decompress:CT_CRYPTO  :FROM:  2,498,433 TO: 10,436,335      258 MS

Decompress:CT_MINILZO :FROM:  3,940,072 TO: 10,436,335       42 MS

Decompress:CT_ZLIB    :FROM:  3,299,771 TO: 10,436,335       69 MS

Decompress:CT_BZIP    :FROM:  2,270,695 TO: 10,436,335      390 MS

 

As expected, MINILZO is the fastest.  Unfortunately MINILZO uses a GPL
license and cannot be used in commercial products.  The rest have licenses
which are more flexible.  Check the links for each package to see if it is
right for you.  I did include the 'CryptoPP' library for completeness though
but, to be frank, I am not very impressed with this package as the
compression and decompression rates are very poor.  As you would expect BZIP
gets the best compression rate but does not perform as quickly.

 

I am fairly impressed with ZLIB, especially because you can use it in a
streaming mode as well, which is excellent for network communications
layers.  (FYI, the assembly language version of the ZLIB decompressor is
hardly any faster than the optimized C code and was not included here.)

 

You might wonder why I'm even bothering to post this little snippet.  Beside
the fact that I hope to save other programmers a little bit of time in the
future, I encourage any other developers of compression libraries to drop
them into this framework so we can stop comparing apples to oranges when it
comes to these technologies.  A large XML file is a typical use case for the
kind of data a game developer might want to squeeze out some space savings
for.  I am also happy to modify the installer and include more standardized
sample data if that is relevant.

 

I started to add in support for the LZMA library, however they only offer a
simple sample decompressor while the compression code is a lot more
difficult to extract into a single C style API call.

 

This demo comes with an easy to use installer and a solution and project
file for visual studio 2005.  All of the compression code itself is
multi-platform and only the little demo app makes any OS specific calls.
The libraries are each included as raw source, each located in their own
directory.  None of the source has been removed (except for test code) so
you are not required to use the compression libraries via the wrapper layer.

 

If anyone feels compelled to add additional compression libraries to this
test framework please let me know and I will make a point to include it in a
new drop.

 

Thanks, I hope somebody finds this useful.

 

 

John