Re: [Numpy-discussion] saving data: which format is recommended?

[Francesc A. <fa...@ca...>]

El dl 22 de 05 del 2006 a les 12:49 -0400, en/na Brian Blais va
escriure:
> I am trying to save numpy arrays (actually a list of them) for later use,=
 and
> distribution to others.  Up until yesterday, I've been using the zpickle =
module from
> the Cookbook, which is just pickle binary format with gzip compression.  =
Yesterday, I
> upgraded my operating system, and now I can't read those files.  I am usi=
ng numpy
> 0.9.9.2536, and unfortunately I can't recall the version that I was using=
, but it was
> pretty relatively recent.  I also upgraded from Python 2.3 to 2.4.  Tryin=
g to load
> the "old" files, I get:
>=20
> AttributeError: 'module' object has no attribute 'dtypedescr'
>=20
>=20
> the file consists of a single dictionary, with two elements, like:
>=20
> var=3D{'im': numpy.zeros((5,5)),'im_scale_shift':[0.0,1.0]}

This could be because NumPy objects has suffered some changes in their
structure in the last months. After 1.0 version there will (hopefully)
be no more changes in the structure, so your pickles will be more stable
(but again, you might have problems in the long run, i.e. when NumPy 2.0
will appear).

>=20
> My question isn't how can I load these "old" files, because I can regener=
ate them.  I
> would like to know what file format I should be using so that I don't hav=
e to worry
> about upgrades/version differences when I want to load them.  Is there a =
preferred
> way to do this?  I thought pickle was that way, but perhaps I don't under=
stand how
> pickle works.

If you need full comaptibility, a better approach than pickle-based
solutions are the .tofile() .fromfile() methods, but you need to save
the metadata for your objects (type, shape, etc.) separately.

If you need full support for saving data & metadata for your NumPy
objects in a transparent way that is independent of pickle you may want
to have a look at PyTables [1] or NetCDF4 [2]. Both packages should be
able to save NumPy datasets without a need to worry for future changes
in NumPy data structures. These both packages are ultimately based on
the HDF5 format[3], which has a pretty strong commitement with
backward/forward format compatibility along its versions.

[1]http://www.pytables.org
[2]http://www.cdc.noaa.gov/people/jeffrey.s.whitaker/python/netCDF4.html=20
[3]http://hdf.ncsa.uiuc.edu/HDF5

Cheers,

--=20
>0,0<   Francesc Altet     http://www.carabos.com/
V   V   C=E1rabos Coop. V.   Enjoy Data
 "-"