Re: [Pytables-users] Chunk selection for optimized data access

[Tim B. <tim...@ma...>]

I was playing around with in-memory HDF5 prior to the 3.0 release. Here's an example based on what I was doing.
I looked over the docs and it does mention that there is an option to throw away the 'file' rather than write it to disk.
Not sure how to do that and can't actually think of a use case where I would want to :-)

And be wary, it is H5FD_CORE.


On Jun 05, 2013, at 08:38 AM, Anthony Scopatz <sc...@gm...> wrote:

I think that you want to set parameters.DRIVER to H5DF_CORE [1].  I haven't ever used this personally, but it would be great to have an example script, if someone wants to write one ;)

 

import numpy as np
import tables

CHUNKY = 30 
CHUNKX = 8640

if __name__ == '__main__':

	
    # create dataset and add global attrs

    file_path = 'demofile_chunk%sx%d.h5' % (CHUNKY, CHUNKX)

    with tables.open_file(file_path, 'w', title='PyTables HDF5 In-memory example', driver='H5FD_CORE') as h5f:
        
        # dummy some data
        lats = np.empty([4320])
        lons = np.empty([8640])

        # create some simple arrays
        lat_node = h5f.create_array('/', 'lat', lats, title='latitude')
        lon_node = h5f.create_array('/', 'lon', lons, title='longitude')

        # create a 365 x 4320 x 8640 CArray of 32bit float
        shape = (365, 4320, 8640)
        atom = tables.Float32Atom(dflt=np.nan)

        # chunk into daily slices and then further chunk days
        sst_node = h5f.create_carray(h5f.root, 'sst', atom, shape, chunkshape=(1, CHUNKY, CHUNKX))

        # dummy up an ndarray
        sst = np.empty([4320, 8640], dtype=np.float32)
        sst.fill(30.0)

        # write ndarray to a 2D plane in the HDF5
        sst_node[0] = sst