Re: [Pytables-users] Chunk selection for optimized data access

[Anthony S. <sc...@gm...>]

Thanks Antonio and Tim!

These are great. I think that one of these should definitely make it into
the examples/ dir.

Be Well
Anthony


On Wed, Jun 5, 2013 at 8:10 AM, Francesc Alted <fa...@gm...> wrote:

> On 6/5/13 11:45 AM, Andreas Hilboll wrote:
> > On 05.06.2013 10:31, Andreas Hilboll wrote:
> >> On 05.06.2013 03:29, Tim Burgess wrote:
> >>> I was playing around with in-memory HDF5 prior to the 3.0 release.
> >>> Here's an example based on what I was doing.
> >>> I looked over the docs and it does mention that there is an option to
> >>> throw away the 'file' rather than write it to disk.
> >>> Not sure how to do that and can't actually think of a use case where I
> >>> would want to :-)
> >>>
> >>> And be wary, it is H5FD_CORE.
> >>>
> >>>
> >>> On Jun 05, 2013, at 08:38 AM, Anthony Scopatz <sc...@gm...>
> wrote:
> >>>> I think that you want to set parameters.DRIVER to H5DF_CORE [1].  I
> >>>> haven't ever used this personally, but it would be great to have an
> >>>> example script, if someone wants to write one ;)
> >>>>
> >>>
> >>>
> >>> import numpy as np
> >>> import tables
> >>>
> >>> CHUNKY = 30
> >>> CHUNKX = 8640
> >>>
> >>> if __name__ == '__main__':
> >>>
> >>>      # create dataset and add global attrs
> >>>
> >>>      file_path = 'demofile_chunk%sx%d.h5' % (CHUNKY, CHUNKX)
> >>>
> >>>      with tables.open_file(file_path, 'w', title='PyTables HDF5
> In-memory
> >>> example', driver='H5FD_CORE') as h5f:
> >>>
> >>>          # dummy some data
> >>>          lats = np.empty([4320])
> >>>          lons = np.empty([8640])
> >>>
> >>>          # create some simple arrays
> >>>          lat_node = h5f.create_array('/', 'lat', lats,
> title='latitude')
> >>>          lon_node = h5f.create_array('/', 'lon', lons,
> title='longitude')
> >>>
> >>>          # create a 365 x 4320 x 8640 CArray of 32bit float
> >>>          shape = (365, 4320, 8640)
> >>>          atom = tables.Float32Atom(dflt=np.nan)
> >>>
> >>>          # chunk into daily slices and then further chunk days
> >>>          sst_node = h5f.create_carray(h5f.root, 'sst', atom, shape,
> >>> chunkshape=(1, CHUNKY, CHUNKX))
> >>>
> >>>          # dummy up an ndarray
> >>>          sst = np.empty([4320, 8640], dtype=np.float32)
> >>>          sst.fill(30.0)
> >>>
> >>>          # write ndarray to a 2D plane in the HDF5
> >>>          sst_node[0] = sst
> >> Thanks Tim,
> >>
> >> I adapted your example for my use case (I'm using the EArray class,
> >> because I need to continuously update my database), and it works well.
> >>
> >> However, when I use this with my own data (but also creating the arrays
> >> like you did), I'm running into errors like "Could not wait on barrier".
> >> It seems like the HDF library is spawing several threads.
> >>
> >> Any idea what's going wrong? Can I somehow avoid HDF5 multithreading at
> >> runtime?
> > Update:
> >
> > When setting max_blosc_threads=2 and max_numexpr_threads=2, everything
> > seems to work as expected (but a bit on the slow side ...).
>
> BTW, can you really notice the difference between using 1, 2 or 4
> threads?  Can you show some figures?  Just curious.
>
> --
> Francesc Alted
>
>
>
> ------------------------------------------------------------------------------
> How ServiceNow helps IT people transform IT departments:
> 1. A cloud service to automate IT design, transition and operations
> 2. Dashboards that offer high-level views of enterprise services
> 3. A single system of record for all IT processes
> http://p.sf.net/sfu/servicenow-d2d-j
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>