Re: [Algorithms] Comparing apples to apples : compression libraries
Brought to you by:
vexxed72
Menu
▾
▴
Re: [Algorithms] Comparing apples to apples : compression libraries
|
From: cat s. <ga...@gm...> - 2008-03-26 01:03:10
|
Can it support a 7-ZIP algorithm? 2008/3/26, John W. Ratcliff <jo...@si...>: > > With pretty much every developer there comes a time when you need to use > a compression library; not because you want to 'zip' up a massive archive > but more because you want to compress some game data on the fly before you > transmit it over the network, or to shrink the size of a local cache or > something. And, like pretty much every developer, you end up wandering > through a maze of open source libraries on the internet, each of which has > entirely different API, source code layout, and license agreement. Just > figuring out to call 'compressData' (if you even can) is often hours of > wading through documentation and dealing with build/configuration issues. > > > > For your convenience I am releasing a small code snippet that I wrote when > I was trying to evaluate a replacement compression library than the one we > are currently using. The simple fact of the matter is that I don't believe > any other programmer should have to waste the same amount of stupid time I > did wrestling these libraries into a convenient form. > > > > http://www.amillionpixels.us/test_compression_1.0.exe > > > > > > Here is the complete API to my wrapper interface: > > > > // Block compression/decompression library wrapper by John W. Ratcliff > http://www.codesuppository.blogspot.com/ > > > > enum CompressionType > > { > > CT_INVALID, > > CT_CRYPTO_GZIP, // The CryptoPP library implementation of GZIP > http://www.cryptopp.com > > CT_MINILZO, // The MiniLZO library > http://www.oberhumer.com/opensource/lzo/ > > CT_ZLIB, // The ZLIB library http://www.zlib.net/ > > CT_BZIP, // The BZIP library http://www.bzip.org/ > > }; > > > > void * compressData(const void *source,int len,int &outlen, > CompressionType type=CT_ZLIB); > > void * decompressData(const void *source,int clen,int &outlen); > > void deleteData(void* mem); > > > > CompressionType getCompressionType(const void *mem,int len); > > const char *getCompressionTypeString(CompressionType type); > > > > }; > > > > This release supports four open source compression libraries. The > CRYPTOPP implementation of GZIP, MINILZO, ZLIB, and BZIP. > > > > This API simply supports block memory compression and decompression. You > can compress a block of memory using any one of the four compressors. The > compressed memory has a small header on it that indicates which compressor > was used, the size of the uncompressed memory block, and a CRC so that it > can easily and safely decompressed. > > > > The test application, called 'test_compression', loads a roughly 10mb XML > file and runs it through each compressor and decompressor and measures the > performance characteristcs of each. The results are as follows: > > > > > > Testing Compression rate and speed with various compressors. > > --------------------------------------------------------------- > > Compress:CT_CRYPTO :FROM: 10,436,335 TO: 2,498,433 23% 1,170 MS > > Compress:CT_MINILZO :FROM: 10,436,335 TO: 3,940,072 37% 97 MS > > Compress:CT_ZLIB :FROM: 10,436,335 TO: 3,299,771 31% 157 MS > > Compress:CT_BZIP :FROM: 10,436,335 TO: 2,270,695 21% 1,544 MS > > --------------------------------------------------------------- > > Testing Decompression speed with various decompressors. > > --------------------------------------------------------------- > > Decompress:CT_CRYPTO :FROM: 2,498,433 TO: 10,436,335 258 MS > > Decompress:CT_MINILZO :FROM: 3,940,072 TO: 10,436,335 42 MS > > Decompress:CT_ZLIB :FROM: 3,299,771 TO: 10,436,335 69 MS > > Decompress:CT_BZIP :FROM: 2,270,695 TO: 10,436,335 390 MS > > > > As expected, MINILZO is the fastest. Unfortunately MINILZO uses a GPL > license and cannot be used in commercial products. The rest have licenses > which are more flexible. Check the links for each package to see if it is > right for you. I did include the 'CryptoPP' library for completeness though > but, to be frank, I am not very impressed with this package as the > compression and decompression rates are very poor. As you would expect BZIP > gets the best compression rate but does not perform as quickly. > > > > I am fairly impressed with ZLIB, especially because you can use it in a > streaming mode as well, which is excellent for network communications > layers. (FYI, the assembly language version of the ZLIB decompressor is > hardly any faster than the optimized C code and was not included here.) > > > > You might wonder why I'm even bothering to post this little snippet. > Beside the fact that I hope to save other programmers a little bit of time > in the future, I encourage any other developers of compression libraries to > drop them into this framework so we can stop comparing apples to oranges > when it comes to these technologies. A large XML file is a typical use case > for the kind of data a game developer might want to squeeze out some space > savings for. I am also happy to modify the installer and include more > standardized sample data if that is relevant. > > > > I started to add in support for the LZMA library, however they only offer > a simple sample decompressor while the compression code is a lot more > difficult to extract into a single C style API call. > > > > This demo comes with an easy to use installer and a solution and project > file for visual studio 2005. All of the compression code itself is > multi-platform and only the little demo app makes any OS specific calls. > The libraries are each included as raw source, each located in their own > directory. None of the source has been removed (except for test code) so > you are not required to use the compression libraries via the wrapper layer. > > > > If anyone feels compelled to add additional compression libraries to this > test framework please let me know and I will make a point to include it in a > new drop. > > > > Thanks, I hope somebody finds this useful. > > > > > > John > > ------------------------------------------------------------------------- > Check out the new SourceForge.net Marketplace. > It's the best place to buy or sell services for > just about anything Open Source. > > http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace > _______________________________________________ > GDAlgorithms-list mailing list > GDA...@li... > https://lists.sourceforge.net/lists/listinfo/gdalgorithms-list > Archives: > http://sourceforge.net/mailarchive/forum.php?forum_name=gdalgorithms-list > |
