Re: [Pytables-users] Chunk selection for optimized data access

[Seref A. <ser...@gm...>]

You would be suprised to see how convenient HDF5 can be in small scale data
:) There are cases where one may need to use binary serialization of a few
thousand items, but still needing metadata, indexing and other nice
features provided by HDF5/pyTables.




On Wed, Jun 5, 2013 at 2:29 AM, Tim Burgess <tim...@ma...> wrote:

> I was playing around with in-memory HDF5 prior to the 3.0 release. Here's
> an example based on what I was doing.
> I looked over the docs and it does mention that there is an option to
> throw away the 'file' rather than write it to disk.
> Not sure how to do that and can't actually think of a use case where I
> would want to :-)
>
> And be wary, it is H5FD_CORE.
>
>
> On Jun 05, 2013, at 08:38 AM, Anthony Scopatz <sc...@gm...> wrote:
>
>
> I think that you want to set parameters.DRIVER to H5DF_CORE [1].  I
> haven't ever used this personally, but it would be great to have an example
> script, if someone wants to write one ;)
>
>
>
> import numpy as np
> import tables
>
> CHUNKY = 30
> CHUNKX = 8640
>
> if __name__ == '__main__':
>
>     # create dataset and add global attrs
>
>     file_path = 'demofile_chunk%sx%d.h5' % (CHUNKY, CHUNKX)
>
>     with tables.open_file(file_path, 'w', title='PyTables HDF5 In-memory
> example', driver='H5FD_CORE') as h5f:
>
>         # dummy some data
>         lats = np.empty([4320])
>         lons = np.empty([8640])
>
>         # create some simple arrays
>         lat_node = h5f.create_array('/', 'lat', lats, title='latitude')
>         lon_node = h5f.create_array('/', 'lon', lons, title='longitude')
>
>         # create a 365 x 4320 x 8640 CArray of 32bit float
>         shape = (365, 4320, 8640)
>         atom = tables.Float32Atom(dflt=np.nan)
>
>         # chunk into daily slices and then further chunk days
>         sst_node = h5f.create_carray(h5f.root, 'sst', atom, shape,
> chunkshape=(1, CHUNKY, CHUNKX))
>
>         # dummy up an ndarray
>         sst = np.empty([4320, 8640], dtype=np.float32)
>         sst.fill(30.0)
>
>         # write ndarray to a 2D plane in the HDF5
>         sst_node[0] = sst
>
>
>
> ------------------------------------------------------------------------------
> How ServiceNow helps IT people transform IT departments:
> 1. A cloud service to automate IT design, transition and operations
> 2. Dashboards that offer high-level views of enterprise services
> 3. A single system of record for all IT processes
> http://p.sf.net/sfu/servicenow-d2d-j
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>
>