Re: [Pytables-users] compression in 0.9

[Jeff W. <jef...@no...>]

Francesc Altet wrote:

>I've ended with a new rewrite of EArray._calcBufferSize method, that I'm
>including at the end of this message. Please, play with the different values
>in lines:
>
>        #bufmultfactor = int(1000 * 10) # Conservative value
>        bufmultfactor = int(1000 * 20)  # Medium value
>        #bufmultfactor = int(1000 * 50) # Aggresive value
>        #bufmultfactor = int(1000 * 100) # Very Aggresive value
>
>and tell me your feedback.
>
>  
>
Francesc:  That new version of _calcbuffersize fixed the problem -  
compression ratios with the 'medium' value of bufmultfactor are back to 
what they were in 0.8.1.  Changing to any of the other values has very 
little effect on file size.  Thanks for your prompt attention to this 
problem!

BTW:  I've added a couple more command line options to nctoh5 - the new 
version is at http://whitaker.homeunix.org/~jeff/nctoh5.  The new 
switches are:

 --unpackshort=(0|1) -- unpack short integer variables to float variables
     using scale_factor and add_offset netCDF variable attributes
     (not active by default).
 --quantize=(0|1) -- quantize data to improve compression using
     least_significant_digit netCDF variable attribute (not active by 
default).
     See http://www.cdc.noaa.gov/cdc/conventions/cdc_netcdf_standard.shtml
     for further explanation of what this attribute means.


-Jeff

-- 
Jeffrey S. Whitaker         Phone : (303)497-6313
NOAA/OAR/CDC  R/CDC1        FAX   : (303)497-6449
325 Broadway                Web   : http://www.cdc.noaa.gov/~jsw
Boulder, CO, USA 80305-3328 Office: Skaggs Research Cntr 1D-124