Re: [Pytables-users] Scientific.IO.NetCDF like API for pytables

[Francesc A. <fa...@py...>]

Hi Jeff,

First of all, thank you very much for your module. It seems very
interesting and I'm looking forward to add it to the forthcoming
release of PyTables.

However, there are some issues that should be addressed first.

1.- I've run the test.py against PyTables 1.1.1 and 1.2 and I'm
getting this error (using both numarray 1.3.3 and Numeric 24.0 (b2)):

[snipped...]
bar[0:4:2,0:3] =
[[0 0 0]
 [2 2 2]]
Scientific.IO.NetCDF must be installed to convert to NetCDF

Scientific.IO.NetCDF must be installed to convert from NetCDF
Traceback (most recent call last):
  File "test_netcdf.py", line 96, in ?
    print file2
  File "/home/faltet/PyTables/pytables/trunk/test/NetCDF.py", line 240, in 
__repr__
    len_unlim = int(self.sync())
  File "/home/faltet/PyTables/pytables/trunk/test/NetCDF.py", line 221, in 
sync
    len_max = max(len_unlim_dims)
ValueError: min() or max() arg is an empty sequence

I don't have the Scientific.IO.NetCDF module installed. Is this normal?

2.- When using both numarray 1.4.1 and Numeric 24.0 (no beta), I'm
getting the next error:

[snipped...]
[  0.   1.   0.]
 [  0.   1.   0.]]' cannot be converted into an array object compliant with 
Array:
'/foo (Array(20, 3)) 'foo'
  type = Float32
  stype = 'Float32'
  shape = (20, 3)
  itemsize = 4
  nrows = 20
  flavor = 'Numeric'
  byteorder = 'little''
The error was: <array sizes must be consistent.>

With these versions of Numeric and numarray all the PyTables tests
passes. The error is the same using both PyTables 1.1.1 and 1.2. Could
you look into this?

3.- It seems that tables.NetCDF only supports compression for
extendeable Arrays (EArrays). Are you aware that CArrays also supports
compression without allowing extendeability?

Now, in order to integrate the module better in PyTables it is
strongly suggested to provide a test unit with a few tests in there,
and also some document with a proper description, but also with
examples, if possible, that would be able to be integrated in the
User's Guide as an appendix (or even as a chapter). Would you like to
provide those? If you don't want to learn tbook, it doesn't matter,
you can provide plain text (restructured text, if you want) and I'll
format it into tbook.

Finally, I'm curious about your 'quantize' method. The HDF5 crew will
be introducing a new filter called 'scale-offset' for forthcoming
1.8.x series. From the release notes of HDF5 1.7.51:

    Scale-Offset compression performs a scale and/or offset operation
    on each data value and truncates the resulting value to a minimum
    number of bits and then stores the data.
    Scaleoffset filter supports floating-point and integer datatype.

Do you think this is going to be compatible with quantize? If so, I'll
try to give support for it in PyTables (when HDF5 1.8 whould be out,
of course).

Regards,

A Dimarts 01 Novembre 2005 22:58, Jeff Whitaker va escriure:
> Francesc et al:
>
> I've written a module that emulates the Scientific.IO.NetCDF API, but
> saves the data in a pytables hdf5 file.  Enclosed is a tarball that
> contains:
>
> 1) NetCDF.py is the Scientific.I0.NetCDF emulator for pytables.
> It should be installed in <sys.prefix>/lib/pythonX.Y/site-packages/tables.
> To use, 'import tables.NetCDF'.
>
> 2) test.py is a simple test.script.
>
> 3) h5.py is a plugin for the python OPeNDAP module, available from
> http://opendap.oceanografia.org/. Just drop it in the dap/plugins
> directory before running setup.py. This allows files created with the
> tables.NetCDF API to be served over http with OPenDAP, as long as the
> file extension is h5, H5, hdf5 or HDF5. Note: h5.py is pretty much a
> direct copy of the netcdf plugin (netcdf.py) with only 2 lines changed.
>
> 4) nctoh5.py is a new version of the nctoh5 utility that uses the nctoh5
> method of a tables.NetCDF.NetCDFFile class instance.
>
> Ulimately, I'd like this module to save the data in an hdf5 file that is
> compatible with netCDF 4
> (http://www.unidata.ucar.edu/software/netcdf/netcdf-4/) so that netCDF 4
> clients can read it.  In the meantime, I think some users who are used
> to the Scientific.IO.NetCDF interface might find it useful, especially
> those who are interested in sharing their data over http with OPeNDAP.
>
> -Jeff

-- 
Francesc Altet