[Pytables-users] Re: nctoh5 with lzo compression

[Jeffrey S W. <Jef...@no...>]

Friedemann:  It is possible to improve compression by quantizing the 
data in your netCDF file, then using shuffle + zlib compression when 
converting to h5.  For example, if your data in array 'dat' only has 
useful information in the tenths digit, you can truncate the data using

Numeric.around(scale*dat)/scale

where scale = 2**bits, and
bits is determined so that a precision of 0.1 is retained (in
this case bits=4).

I've found that this can improve compression by up to a factor of 2, if 
you have data with not many significant digits.  This is often the case 
with the meteorological observations I deal with, but your mileage may vary.

BTW: In my experience, shuffle + zlib (level 6 is sufficient) always 
produces the smallest files.

-Jeff

-- 
Jeffrey S. Whitaker         Phone  : (303)497-6313
Meteorologist               FAX    : (303)497-6449
NOAA/OAR/CDC  R/CDC1        Email  : Jef...@no...
325 Broadway                Web    : www.cdc.noaa.gov/~jsw
Boulder, CO, USA 80303-3328 Office : Skaggs Research Cntr 1D-124