We, at Shilps Sciences, design and manufacture AFM and related instruments. We regularly use Gwyddion for our image/data analysis and encourage our customers to do so. A typical AFM scan in our instrument produces 6 to 8 images for various channels measured and common metadata as separate files. To better organize such data we found HDF5 format to be highly suitable. We would also like our files to be directly openable from Gwyddion.
We found that Gwyddion already supports HDF5 files, but do not know the internal structure it expects. We want to know the specific HDF5-Schema (Group/dataset/attributes) that Gwyddion expects so that we can design our software to output the same. Can somebody point us to a resource that gives details of the supported HDF5 format?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Gwyddion can currently import Asylum Research Ergo HDF5 files. Their schema may be formally specified somewhere at Asylum Research, but I only have an informal description, definitely not sufficient to write valid files.
Unless you are going just for the lowest common denominator, trying to coerce data from one manufacturer to a format used by a different one is usually a messy business anyway. If you have already settled on a specific schema for your instrument, it is better if we add support for your HDF5 files. Since some generic HDF5 infrastructure is already there to import Ergo files, adding another HDF5-based data format should not be that difficult.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Thanks for the quick response. We can certainly come up with a new schema suitable for our instrument. We are in the early stages of it. Will get back after finalizing the schema.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
We have come up with a new HDF5 schema for our instrument output data. A sample file is attached and a brief introduction follows.
At the root level we have a group called Session. Inside Session, we have another group called Scan. All the images are created as datasets under the group Scan. So, all the images can be addressed as /Session/Scan/image#. Currently there are 8 images corresponding to 8 channels. All images are two-dimensional arrays of 32-bit floating-point values.
There are various attributes at every level. The channel information is given as attributes to the individual image datasets. Currently there are just three attributes to datasets – Channel, Scan cycle and Units. The combination of Channel and Scan cycle will form a unique identifier for the channel.
Other important attributes are at the Scan group level. The scan length and number of points are given as X Range and X No respectively.
The resolution in the x-direction can be computed as (X Range)/(X No)
Likewise, the resolution in the y-direction can be computed as (Y Range)/(Y No)
All other attributes are instrument specific and can be safely ignored.
As of r25348, your file format should be implemented in the hdf5file module. Please try tomorrow's or later development snapshot of Gwyddion.
Now a few remarks:
What are the units of X|Y Range? Micrometres?
What is the data order/orientation? Gwyddion uses a left-handed coordinate system, so data probably should be reflected around the y axis, but I cannot be sure.
The representation of metadata under /Session is positively weird. Using strings for integers and floats in HDF5 despite its ability to represent integer and float attributes directly is, well, your choice. But the extra spaces at the beginning, extra newlines at the end and quotes in true string attributes look like a relic of hasty rewrite from a text format. I mean, does an empty string really need to be represented as space, quote, quote, new line – as opposted to actual empty string?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Thanks for the quick response and code updates. Below are our responses inline to remarks:
• What are the units of X|Y Range? Micrometres?
Yes, micrometers is correct.
• What is the data order/orientation? Gwyddion uses a left-handed coordinate system, so data probably should be reflected around the y axis, but I cannot be sure.
We can follow the left-handed coordinate system.
• The representation of metadata under /Session is positively weird. Using strings for integers and floats in HDF5 despite its ability to represent integer and float attributes directly is, well, your choice. But the extra spaces at the beginning, extra newlines at the end and quotes in true string attributes look like a relic of hasty rewrite from a text format. I mean, does an empty string really need to be represented as space, quote, quote, new line – as opposted to actual empty string?
The datatype mismatch is a result of us trying to convert our old data and configuration files into HDF5 format. This will be taken care of in the new software which will directly save HDF5 files.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
We have updated our main instrument control software to save HDF files according to the schema now supported by Gwyddion. We are able to open the AFM scans saved as .h5 files with Gwyddion and the images are consistent with what we get in our GUI (please see snapshots attached). We are also able to select the different channels to view with the help of “Data Brower” feature under the Info menu.
We would like to highlight that along with AFM our instrument also supports some advanced optical microscopy techniques such as White Light Interferometry. When the output is 2D data, we follow the same schema as AFM scans to get Gwyddion to open our optical reconstruction (topography) images. But some of these modes generate 3D Volume data, which we are currently unable to open with Gwyddion.
Kindly advise us on how we can get our 3D data to be compatible with Gwyddion. Currently we have a different HDF5 group where we store volume data as a stack of images. We also create sparse XYZ data for reconstructed surfaces (optical tomography).
If you store volume data in HDF5 as a stack of images, that should be straightforward to add. One big cube would also be OK.
As for sparse XYZ data, I am not sure if there is any better HDF5 representation than a simple data set with n×3 elements That would be fine for us.
Both data types are already implemented in Gwyddion (for other file formats). So, it should be basically a matter of providing format/schema specifications (and them being reasonable).
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
For volume data, we define a new group “/Session/Volume/” and the 3D array datasets are stored as Image3d1, Image3d2,…
The Volume group attributes define the resolutions similar to Scan group. Along with “X No”, “X Range”, “Y No”, “Y Range”, we have also added “Z No”, “Z Range”. The Range units are “micrometer” as usual. The dataset attributes are also similar.
For sparse XYZ data, we define a new group “/Session/XYZ/” and the simple nx3 array datasets are stored as Scatter1, Scatter2,…
There are other groups (used for internal purpose) within the sample file which can be ignored for the time being.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Thanks for the update. We are basically in the middle of releasing version 2.63 now. So it will not include support for any of these additional data types; it is too late for that. However, I will look into it afterwards.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Apart from the 3D datasets, another important datatype that we want to be compliant with Gwyddion is the Point Spectroscopy data.
As is typical with several AFMs, our instrument software also allows the user to select a set of points (x,y coordinates) on a scanned image, and perform spectroscopy (basically force-distance curves) on each of those points. Our existing method of storing this data involves a large spreadsheet file recording all the point spectra side-by-side.
We believe storing this kind of associated data can be done much more elegantly with HDF format. Can you provide any guidelines on how such spectral data are typically stored and how they are associated by Gwyddion with the points on the scan image?
Our spectroscopy routine can be configured to sweep voltage instead of distance; essentially measuring IV curves at each point. Also worth noting is that, we normally measure more than one channel (for example, Force and Current) in each Spectroscopy measurement. So, the spectra datasets need to specify the x and y channels as well.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Let's avoid trying to do too many things at once. I am currently adding volume data import and I am confused by what I see in your sample file wli_test_file_0.h5.
The HDF5 dimensions for Image3d1 are ( 160, 200, 600 ) – as reported by H5Sget_simple_extent_dims(), h5dump, etc.
But the resolutions X No, Y No and Z No are 160, 200 and 601.
So, first there is a mismatch 600 vs 601. This probably just means incorrect Z No value.
Furthermore, when reading images, the dimensions go from the outermost (slowest moving) index to the innermost (fastest changing). Both in your and Ergo files. So in the HDF5 dimensions y resolution is first, then x.
Now although HDF5 viewers like https://myhdf5.hdfgroup.org/ show your files reasonably, I do not understand how the dimensions can be ordered x, y then z. Doesn't it make x the slowest moving index?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
You are right about the order of the dimensions: y, x then z. The X No, Y No attributes were interchanged and now corrected.
The Z No value also matches the dataset size.
The Zscan/ImageStack has raw data and is not desired to be used for visualization or analysis in Gwyddion. This raw data is meant to be further processed by another piece of software that generates the Volume data which is what we want the AFM users to see.
All Range units are micrometer for the Profilometry/OCT (Optical Coherence Tomography) applications which we are focussing on. At this point we do not foresee a scenario where the third coordinate is not spatial.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
OK, thank you for the clarification. I have implemented volume and XYZ data import and am looking into Curves.
I am not sure why the dimensions of a curve are 6612 by 41. I guess it is actually 41 different curves? Or 1 abscissa + 40 curves? Also there is only a single unit attribute, presumably for values? Other data types have the range/lateral units specified in the group.
If I may I suggest to unify the range unit attribute names. Currently it seems a bit chaotic:
"Range Units" for images.
"Range units" for volume data.
"Units" (in the group) for XYZ data.
Curves bring us back to spectroscopy. Single point spectroscopy is not handled ideally in Gwyddion, but the data structure is probably fine. SPS data are a set of curves, each with (x,y) coordinates – presumably in the accompanying images. If you measure more curve types at each point, it would mean multiple Spectra objects in Gwyddion, but that may not be a universally good representation…
So in HDF5 you can have each measured location as a dataset in some spectroscopy group. The coordinates would be then dataset attributes. This allows each curve to have a different number of points, etc. For more uniform data, it may be more convenient to lump everything together and add some arrays providing the coordinates. Anything reasonable along these lines would be straightforward to import to Gwyddion.
Finally, please provide an updated HDF5 file for testing after if you make changes – like correcting Z No (or renaming the unit attribute names if you are going to do it).
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
There are 6612 points (in this experiment) where position (x,y,z) is explicitly calculated. These points are kept in /XYZ/Scatter1. The dataset there has shape 6612 x 3. For each of those same 6612 points, a spectra (Power vs wavelength) is estimated. The x axis (wavelength) has 41 points (in this experiment). This data is stored in /Curves/Spectrum1. Hence the shape of the dataset is 6612 x 41. So each spectra in /Curves/Spectrum1 is associated with it's corresponding x,y,z point in /XYZ/Scatter1. In a 3D scatter plot, we use the spectra to colour the points, providing both spatial and spectral information.
We recognize that this is a very specialized application and it is okay if Gwyddion cannot associate a 3D point with spectra.
But for spectral curves associated with regular AFM scans, we would certainly want to make use of Gwyddion capabilities. As you have rightly said, our single point spectroscopy curves do have different lengths. So, it is best to have a separate dataset for each location whose coordinate is stored as dataset attributes. We can group these datasets under /Session/Spectra/
The associated AFM scan image(s) will be under /Session/Scan/. In this type of file there would be only Scan and Spectra groups for the time being. We will provide a sample file based on your inputs.
Please let us know when we can expect an updated exe installer in the nightly snapshot.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I tried to add data from /Curves as single point spectroscopy, but the result is a monstrosity (see screenshot). Gwyddion SPS are meant for a handful of spectra, not six thousand.
The spectra coordinates seem to actually lie on a regular grid; they are just sparse. Such data would probably work better if imported as a curve map to Gwyddion – the same way as a full grid curve map. For curve maps it is OK if data are missing in some points, or in fact in most points. However, it is difficult for the import module to figure out the points are in fact a subset of regular grid. The way curves are represented in your files does not make it manifest.
I am sorry about the nightly snapshots. Our build server is currently broken. We are working on it.
As mentioned in our previous post, the /Session/Curves group contains spectra for points on a 3D space/grid. This data is a remnant of our Optical Coherence Tomography (OCT) measurement/post-processing and may not need to be opened in Gwyddion.
We actually want to utilize the Gwyddion SPS feature on a different group /Session/Spectra, which typically contains force-distance curves for points on regular AFM scans. We have attached a sample file for this data. This file only contains data from AFM scans and associated single point spectroscopy. These Spectra datasets contain four columns which are labelled in the attributes along with point coordinates. The first column has the horizontal-axis variable (Z or voltage), and the next two columns are the vertical-axis variables for the graphs. The last column is always TimeStamp.
We have tested the latest nightly snapshot.
If the hdf file contains Volume data (/Session/Volume/Image3d1), Gwyddion is able to open it. But, if the file contains sparse data (/Session/XYZ/Scatter1) with or without spectrum (/Session/Curves/Spectrum1), then Gwyddion does not load the file; gives the following error:
Invalid field dimension: 228298.
It is okay if Gwyddion ignores the /Session/Curves group completely. But it should be able to open the other datasets, especially the 2D images in the same file.
As mentioned earlier we are more interested in the Gwyddion SPS feature to open datasets in a different group /Session/Spectra, which typically contains force-distance curves for points on regular AFM scans. A sample file for this data is attached in my previous post.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
The invalid field dimensions error was a mistake. fixed now in r25561. It should have never produced this error.
The import module is still in development. Currently it leans towards rather strict error checking. We may modify it later to try to read at least something even if some bits of data do not seem valid.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
We have tested the recent nightly snapshots. Thanks for the updates that allowed us to open our HDF files.
One point about the Gwyddion SPS feature that opens the datasets from the group /Session/Spectra. These Spectra datasets contain four columns - Z, Amplitude, Force, and TimeStamp. Gwyddion is plotting each column separately against an index in the x-axis. How do we plot Amplitude vs Z or Force vs Z? Is there any option in Gwyddion or do we need to change the way we store data?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Spectra (and graphs) in Gwyddion have fixed roles of abscissa and ordinate. Meaning some quantity is on the x axis, some other on the y axis and it is not easy (or in fact even possible in the GUI) to change it. This limitation is a consequence of how data the structures were designed and we just have to deal with it.
The import module can recognise ‘this quantity should be abscissa’ and ‘this quantity should be ordinate’ and create the corresponding plots. This has to be done explicitly during the import. No such logic is currently there because I did not know what it should really do.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
We, at Shilps Sciences, design and manufacture AFM and related instruments. We regularly use Gwyddion for our image/data analysis and encourage our customers to do so. A typical AFM scan in our instrument produces 6 to 8 images for various channels measured and common metadata as separate files. To better organize such data we found HDF5 format to be highly suitable. We would also like our files to be directly openable from Gwyddion.
We found that Gwyddion already supports HDF5 files, but do not know the internal structure it expects. We want to know the specific HDF5-Schema (Group/dataset/attributes) that Gwyddion expects so that we can design our software to output the same. Can somebody point us to a resource that gives details of the supported HDF5 format?
Gwyddion can currently import Asylum Research Ergo HDF5 files. Their schema may be formally specified somewhere at Asylum Research, but I only have an informal description, definitely not sufficient to write valid files.
Unless you are going just for the lowest common denominator, trying to coerce data from one manufacturer to a format used by a different one is usually a messy business anyway. If you have already settled on a specific schema for your instrument, it is better if we add support for your HDF5 files. Since some generic HDF5 infrastructure is already there to import Ergo files, adding another HDF5-based data format should not be that difficult.
Thanks for the quick response. We can certainly come up with a new schema suitable for our instrument. We are in the early stages of it. Will get back after finalizing the schema.
We have come up with a new HDF5 schema for our instrument output data. A sample file is attached and a brief introduction follows.
At the root level we have a group called Session. Inside Session, we have another group called Scan. All the images are created as datasets under the group Scan. So, all the images can be addressed as /Session/Scan/image#. Currently there are 8 images corresponding to 8 channels. All images are two-dimensional arrays of 32-bit floating-point values.
There are various attributes at every level. The channel information is given as attributes to the individual image datasets. Currently there are just three attributes to datasets – Channel, Scan cycle and Units. The combination of Channel and Scan cycle will form a unique identifier for the channel.
Other important attributes are at the Scan group level. The scan length and number of points are given as X Range and X No respectively.
The resolution in the x-direction can be computed as (X Range)/(X No)
Likewise, the resolution in the y-direction can be computed as (Y Range)/(Y No)
All other attributes are instrument specific and can be safely ignored.
As of r25348, your file format should be implemented in the hdf5file module. Please try tomorrow's or later development snapshot of Gwyddion.
Now a few remarks:
What are the units of X|Y Range? Micrometres?
What is the data order/orientation? Gwyddion uses a left-handed coordinate system, so data probably should be reflected around the y axis, but I cannot be sure.
The representation of metadata under /Session is positively weird. Using strings for integers and floats in HDF5 despite its ability to represent integer and float attributes directly is, well, your choice. But the extra spaces at the beginning, extra newlines at the end and quotes in true string attributes look like a relic of hasty rewrite from a text format. I mean, does an empty string really need to be represented as space, quote, quote, new line – as opposted to actual empty string?
Thanks for the quick response and code updates. Below are our responses inline to remarks:
• What are the units of X|Y Range? Micrometres?
Yes, micrometers is correct.
• What is the data order/orientation? Gwyddion uses a left-handed coordinate system, so data probably should be reflected around the y axis, but I cannot be sure.
We can follow the left-handed coordinate system.
• The representation of metadata under /Session is positively weird. Using strings for integers and floats in HDF5 despite its ability to represent integer and float attributes directly is, well, your choice. But the extra spaces at the beginning, extra newlines at the end and quotes in true string attributes look like a relic of hasty rewrite from a text format. I mean, does an empty string really need to be represented as space, quote, quote, new line – as opposted to actual empty string?
The datatype mismatch is a result of us trying to convert our old data and configuration files into HDF5 format. This will be taken care of in the new software which will directly save HDF5 files.
In any case it would be useful to have a verification what the data look like correctly. Perhaps a screenshot?
Then let us know when you update the software/format so that the import module can be updated accordingly.
We have updated our main instrument control software to save HDF files according to the schema now supported by Gwyddion. We are able to open the AFM scans saved as .h5 files with Gwyddion and the images are consistent with what we get in our GUI (please see snapshots attached). We are also able to select the different channels to view with the help of “Data Brower” feature under the Info menu.
We would like to highlight that along with AFM our instrument also supports some advanced optical microscopy techniques such as White Light Interferometry. When the output is 2D data, we follow the same schema as AFM scans to get Gwyddion to open our optical reconstruction (topography) images. But some of these modes generate 3D Volume data, which we are currently unable to open with Gwyddion.
Kindly advise us on how we can get our 3D data to be compatible with Gwyddion. Currently we have a different HDF5 group where we store volume data as a stack of images. We also create sparse XYZ data for reconstructed surfaces (optical tomography).
If you store volume data in HDF5 as a stack of images, that should be straightforward to add. One big cube would also be OK.
As for sparse XYZ data, I am not sure if there is any better HDF5 representation than a simple data set with n×3 elements That would be fine for us.
Both data types are already implemented in Gwyddion (for other file formats). So, it should be basically a matter of providing format/schema specifications (and them being reasonable).
We have amended our HDF Schema to include 3D datasets and tried to follow Gwyddion terminology as much as possible. A sample file is given in the following link.
https://drive.google.com/file/d/1qfGTcq42M4Wb5j9vPDr_XauKFNswF9SR/view?usp=sharing
For volume data, we define a new group “/Session/Volume/” and the 3D array datasets are stored as Image3d1, Image3d2,…
The Volume group attributes define the resolutions similar to Scan group. Along with “X No”, “X Range”, “Y No”, “Y Range”, we have also added “Z No”, “Z Range”. The Range units are “micrometer” as usual. The dataset attributes are also similar.
For sparse XYZ data, we define a new group “/Session/XYZ/” and the simple nx3 array datasets are stored as Scatter1, Scatter2,…
There are other groups (used for internal purpose) within the sample file which can be ignored for the time being.
Thanks for the update. We are basically in the middle of releasing version 2.63 now. So it will not include support for any of these additional data types; it is too late for that. However, I will look into it afterwards.
Apart from the 3D datasets, another important datatype that we want to be compliant with Gwyddion is the Point Spectroscopy data.
As is typical with several AFMs, our instrument software also allows the user to select a set of points (x,y coordinates) on a scanned image, and perform spectroscopy (basically force-distance curves) on each of those points. Our existing method of storing this data involves a large spreadsheet file recording all the point spectra side-by-side.
We believe storing this kind of associated data can be done much more elegantly with HDF format. Can you provide any guidelines on how such spectral data are typically stored and how they are associated by Gwyddion with the points on the scan image?
Our spectroscopy routine can be configured to sweep voltage instead of distance; essentially measuring IV curves at each point. Also worth noting is that, we normally measure more than one channel (for example, Force and Current) in each Spectroscopy measurement. So, the spectra datasets need to specify the x and y channels as well.
Let's avoid trying to do too many things at once. I am currently adding volume data import and I am confused by what I see in your sample file
wli_test_file_0.h5.The HDF5 dimensions for Image3d1 are ( 160, 200, 600 ) – as reported by H5Sget_simple_extent_dims(), h5dump, etc.
But the resolutions X No, Y No and Z No are 160, 200 and 601.
So, first there is a mismatch 600 vs 601. This probably just means incorrect Z No value.
Furthermore, when reading images, the dimensions go from the outermost (slowest moving) index to the innermost (fastest changing). Both in your and Ergo files. So in the HDF5 dimensions y resolution is first, then x.
Now although HDF5 viewers like https://myhdf5.hdfgroup.org/ show your files reasonably, I do not understand how the dimensions can be ordered x, y then z. Doesn't it make x the slowest moving index?
Update: Resolution 601 is correct for ImageStack in group ZScan. This is, however, a different group than than Volume where Z No is given as 601.
Also, what are Range units in Volume? Units of all three X, Y and Z? What if the third coordinate is not spatial, but for instance voltage?
I do not need entirely formal specification, but some description what is what would be helpful.
We have corrected the inconsistencies in the sample file and uploaded it here:
https://drive.google.com/file/d/1qfGTcq42M4Wb5j9vPDr_XauKFNswF9SR/view?usp=sharing
You are right about the order of the dimensions: y, x then z. The X No, Y No attributes were interchanged and now corrected.
The Z No value also matches the dataset size.
The Zscan/ImageStack has raw data and is not desired to be used for visualization or analysis in Gwyddion. This raw data is meant to be further processed by another piece of software that generates the Volume data which is what we want the AFM users to see.
All Range units are micrometer for the Profilometry/OCT (Optical Coherence Tomography) applications which we are focussing on. At this point we do not foresee a scenario where the third coordinate is not spatial.
OK, thank you for the clarification. I have implemented volume and XYZ data import and am looking into Curves.
I am not sure why the dimensions of a curve are 6612 by 41. I guess it is actually 41 different curves? Or 1 abscissa + 40 curves? Also there is only a single unit attribute, presumably for values? Other data types have the range/lateral units specified in the group.
If I may I suggest to unify the range unit attribute names. Currently it seems a bit chaotic:
Curves bring us back to spectroscopy. Single point spectroscopy is not handled ideally in Gwyddion, but the data structure is probably fine. SPS data are a set of curves, each with (x,y) coordinates – presumably in the accompanying images. If you measure more curve types at each point, it would mean multiple Spectra objects in Gwyddion, but that may not be a universally good representation…
So in HDF5 you can have each measured location as a dataset in some spectroscopy group. The coordinates would be then dataset attributes. This allows each curve to have a different number of points, etc. For more uniform data, it may be more convenient to lump everything together and add some arrays providing the coordinates. Anything reasonable along these lines would be straightforward to import to Gwyddion.
Finally, please provide an updated HDF5 file for testing after if you make changes – like correcting Z No (or renaming the unit attribute names if you are going to do it).
We have made 'Range Units' uniform wherever we are referring to Range values (X Range , Y Range or Z Range). But the 'Units' elsewhere refer to dataset values.
Updated sample file is uploaded:
https://drive.google.com/file/d/1ecRF0NZA9E8h5R5OZWNQ9O4k5JB2zNGW/view?usp=sharing
There are 6612 points (in this experiment) where position (x,y,z) is explicitly calculated. These points are kept in /XYZ/Scatter1. The dataset there has shape 6612 x 3. For each of those same 6612 points, a spectra (Power vs wavelength) is estimated. The x axis (wavelength) has 41 points (in this experiment). This data is stored in /Curves/Spectrum1. Hence the shape of the dataset is 6612 x 41. So each spectra in /Curves/Spectrum1 is associated with it's corresponding x,y,z point in /XYZ/Scatter1. In a 3D scatter plot, we use the spectra to colour the points, providing both spatial and spectral information.
We recognize that this is a very specialized application and it is okay if Gwyddion cannot associate a 3D point with spectra.
But for spectral curves associated with regular AFM scans, we would certainly want to make use of Gwyddion capabilities. As you have rightly said, our single point spectroscopy curves do have different lengths. So, it is best to have a separate dataset for each location whose coordinate is stored as dataset attributes. We can group these datasets under /Session/Spectra/
The associated AFM scan image(s) will be under /Session/Scan/. In this type of file there would be only Scan and Spectra groups for the time being. We will provide a sample file based on your inputs.
Please let us know when we can expect an updated exe installer in the nightly snapshot.
I tried to add data from /Curves as single point spectroscopy, but the result is a monstrosity (see screenshot). Gwyddion SPS are meant for a handful of spectra, not six thousand.
The spectra coordinates seem to actually lie on a regular grid; they are just sparse. Such data would probably work better if imported as a curve map to Gwyddion – the same way as a full grid curve map. For curve maps it is OK if data are missing in some points, or in fact in most points. However, it is difficult for the import module to figure out the points are in fact a subset of regular grid. The way curves are represented in your files does not make it manifest.
I am sorry about the nightly snapshots. Our build server is currently broken. We are working on it.
As mentioned in our previous post, the /Session/Curves group contains spectra for points on a 3D space/grid. This data is a remnant of our Optical Coherence Tomography (OCT) measurement/post-processing and may not need to be opened in Gwyddion.
We actually want to utilize the Gwyddion SPS feature on a different group /Session/Spectra, which typically contains force-distance curves for points on regular AFM scans. We have attached a sample file for this data. This file only contains data from AFM scans and associated single point spectroscopy. These Spectra datasets contain four columns which are labelled in the attributes along with point coordinates. The first column has the horizontal-axis variable (Z or voltage), and the next two columns are the vertical-axis variables for the graphs. The last column is always TimeStamp.
FYI, regular night builds should be working again.
We have tested the latest nightly snapshot.
If the hdf file contains Volume data (/Session/Volume/Image3d1), Gwyddion is able to open it. But, if the file contains sparse data (/Session/XYZ/Scatter1) with or without spectrum (/Session/Curves/Spectrum1), then Gwyddion does not load the file; gives the following error:
Invalid field dimension: 228298.
It is okay if Gwyddion ignores the /Session/Curves group completely. But it should be able to open the other datasets, especially the 2D images in the same file.
As mentioned earlier we are more interested in the Gwyddion SPS feature to open datasets in a different group /Session/Spectra, which typically contains force-distance curves for points on regular AFM scans. A sample file for this data is attached in my previous post.
The invalid field dimensions error was a mistake. fixed now in r25561. It should have never produced this error.
The import module is still in development. Currently it leans towards rather strict error checking. We may modify it later to try to read at least something even if some bits of data do not seem valid.
We have tested the recent nightly snapshots. Thanks for the updates that allowed us to open our HDF files.
One point about the Gwyddion SPS feature that opens the datasets from the group /Session/Spectra. These Spectra datasets contain four columns - Z, Amplitude, Force, and TimeStamp. Gwyddion is plotting each column separately against an index in the x-axis. How do we plot Amplitude vs Z or Force vs Z? Is there any option in Gwyddion or do we need to change the way we store data?
Spectra (and graphs) in Gwyddion have fixed roles of abscissa and ordinate. Meaning some quantity is on the x axis, some other on the y axis and it is not easy (or in fact even possible in the GUI) to change it. This limitation is a consequence of how data the structures were designed and we just have to deal with it.
The import module can recognise ‘this quantity should be abscissa’ and ‘this quantity should be ordinate’ and create the corresponding plots. This has to be done explicitly during the import. No such logic is currently there because I did not know what it should really do.