We want to use hdf5 to store (possibly quite large) multi-dimension data sets. We're acquiring waveforms from an oscilloscope and changing a configurable number of variables (or dimensions) per waveform. For example, we might be acquiring these waveforms having our detector at points along a line in space, or points on a plane or points in 3D etc and we're hoping to store the data in a data set with the same number of dimensions.
The storing bit works fine, we have tested with multi-dimension arrays up to 5 or 6 dimensions and that's all good.
The issue is in reading back. We will rarely want to read all of the data back together. We're more likely to want to read either all the data from one dimension of the dataset (typically 1 full waveform for example), or all of the data from 2 dimensions of the data set (e.g. all waveforms from points along a line) and so on. In our early testing it was quite straightforward to define a part of the dataspace to read back and get that. But in that case we knew the size of our dataset before hand (i.e. how many dimensions etc) and coded to match. We want this code to be flexible and work with any number of dimensions in the data set.
The issue is shown in the attached image - we can successfully read back a single dimension from a 3D data set by setting up slab appropriately and then pasing that to the read data xnode, and wiring an empty 1D array to the "default" input. BUT, when we try to extend this to requesting a 2D slice, we get an error saying the data from h5 is 3 dimensions... Is this a bug? Or are we doing something wrong?
Hi Paul,
Sorry again for missing this - thanks for the descriptive scenario; it's not something I had considered myself previously. The dimensionality restriction was originally applied for sanity-checking in the C library because there was a lot of complexity on the LV side for higher-dimensional arrays. The implementation means you can always perform a read as either a "flat" 1D array (irrespective of the stored dimensionality), or as an array of equal dimension to the stored array.
So for this case of extracting a 2D slice from a 3D array, I would have suggested reading back into a 1D array and then using "Reshape Array" to convert to 2D, or reading back into a 3D array and using "Index Array" to drop the unnecessary axis.
I think it would be viable to modify the C code to automatically squash dimensions for which the associated "count" is 1, which would provide direct handling of this scenario. Or I could drop the dimensionality check altogether and shift the onus on the user to ensure the dimensionality is valid, since there is already a check that the total number of datapoints is unchanged.
Cheers,
Martijn
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
We want to use hdf5 to store (possibly quite large) multi-dimension data sets. We're acquiring waveforms from an oscilloscope and changing a configurable number of variables (or dimensions) per waveform. For example, we might be acquiring these waveforms having our detector at points along a line in space, or points on a plane or points in 3D etc and we're hoping to store the data in a data set with the same number of dimensions.
The storing bit works fine, we have tested with multi-dimension arrays up to 5 or 6 dimensions and that's all good.
The issue is in reading back. We will rarely want to read all of the data back together. We're more likely to want to read either all the data from one dimension of the dataset (typically 1 full waveform for example), or all of the data from 2 dimensions of the data set (e.g. all waveforms from points along a line) and so on. In our early testing it was quite straightforward to define a part of the dataspace to read back and get that. But in that case we knew the size of our dataset before hand (i.e. how many dimensions etc) and coded to match. We want this code to be flexible and work with any number of dimensions in the data set.
The issue is shown in the attached image - we can successfully read back a single dimension from a 3D data set by setting up slab appropriately and then pasing that to the read data xnode, and wiring an empty 1D array to the "default" input. BUT, when we try to extend this to requesting a 2D slice, we get an error saying the data from h5 is 3 dimensions... Is this a bug? Or are we doing something wrong?
Thanks!
Hi Paul,
Sorry again for missing this - thanks for the descriptive scenario; it's not something I had considered myself previously. The dimensionality restriction was originally applied for sanity-checking in the C library because there was a lot of complexity on the LV side for higher-dimensional arrays. The implementation means you can always perform a read as either a "flat" 1D array (irrespective of the stored dimensionality), or as an array of equal dimension to the stored array.
So for this case of extracting a 2D slice from a 3D array, I would have suggested reading back into a 1D array and then using "Reshape Array" to convert to 2D, or reading back into a 3D array and using "Index Array" to drop the unnecessary axis.
I think it would be viable to modify the C code to automatically squash dimensions for which the associated "count" is 1, which would provide direct handling of this scenario. Or I could drop the dimensionality check altogether and shift the onus on the user to ensure the dimensionality is valid, since there is already a check that the total number of datapoints is unchanged.
Cheers,
Martijn
Hi Martijn,
Thanks for the response - we did indeed end up using the reading as a 1D and then reshaping in the end and found that worked well.
Thanks!