In case it is useful for someone else, I've written a Python package to help read and parse gxsm scan files.
Why do this?
The huge flexibility of gxsm means that your ADC channels could correspond to anything. Since gxsm doesn't need to know the particulars of this (except perhaps for the nAmpere2Volt and dHertz2Volt parameters in the gxsm config), it does not request them. In turn, it saves the scanned data as ADC channels in ADC counter units. In short, it lets you interpret the data it saved, based on your SPM configuration.
This means you will likely be converting the data to physical units when interpreting them.
gxsm saves each scan channel in its own NetCDF3 file. It may be of interest to bundle all the channels of a given file together, for analysis purposes.
gxsmread is a package I wrote to help me do those 2 things. For (1), you can provide it a config file, indicating the conversion factor of each channel, units, and a name to give it; it will do the conversion for you. For (2), you can give it the path to a multifile dataset, and it will merge them all together.
None of this is particularly complicated, but not having to write it up each time may save others some time.
Nick,
you brought up an very old -- going back to the roots of gxsm -- quest here also:
The pro and cons of storing channels in separated files! I am kind of glad you did. And wonder if any one else has some input here?
Facts:
* all "meta data" (minor from size for larger scans) besides the Z unit and 4D data set itself is a identical copy. Few exceptions are potentially attached user&probe data sets only been attached to the "master" scan, usually the 1st in line or Topo.
* File name convention, "channel name" and "Direction Mode" suffix to name.
* Any number scan channels, directions, etc.
* Easy channel data preview/selection via system file viewer via thumbnailing.
* Easy selective loading of particular only data channels for analysis.
* Easy data management or later removal of meaning less/obsolete data when forgotten to disable when monitoring additional channels at that time not used. Example left on dFreq/Exec data channels and went back to STM, or a like.
* Many files to manage... -- but on average 2..4 channels or 4x more files is not that big of an deal today.
any further points?
PS: I can -- very easy -- in future imagine for Gxsm4 to have the user choose and allow to store all channels withing one file. Anyways that will break the long back and forward compatibility of the Gxsm NetCDF format -- or will make additional channels inaccessible for older software! However, for Gxsm4 a move to the latest NetCDF4 (or5??) is in the to-do list anyways. It's only a tedious task to port from all the old netcdf lib code to the new... BTW -- nay takers to help?
Last edit: Percy Zahl 2023-10-20
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
PS: a few basic assignments are possible via preferences so. See screenshot.
It it helps, I could make custom per channel conversion factors to a core set of units available in future. For sure you can since ever "replace" the default names ADC1,2,... with any meaning full name there. And the file names will also have that post-fix name.
Thanks for the quick reply! Of course, I am fairly new and did not realize there is this level of control in DataAq. I was dealing with old files that had presumably not been set up in this way before acquisition.
Out of curiosity: what does changing parameters in DataAq do? I assume changing the 'name' (e.g. DataAq/DataSrcA1) changes the saved filename, but what about Zunit and Zlabel? Of the files I have looked at (possibly outdated), all files have data stored in FloatField, and i don't recall there being units or labels.
Thank you,
Nick
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
you must keep the "FloatField" type, as this is a hardware /HwI design requirement.
you can change name and label to anything, may avoid special characters. Like the default exanple shown for ADC0-ITunnel. (No need for ADC0, but I recommend to keep it and add info like "ADC3-TestSignal" so you later know where is was originated from, no need so!)
Label is used for the profile axis label (X or Y as used). Name is the long name is used also for the channel's file name post fix.
You can pick a unit from the lists, but the conversion available right now are limited to "length" Z (Ang/nm/...mm), Current (nA conversion) or Force (nN conversion) and the default "Volts".
"Z" (Z is historically at image pixel value's axis name) and the "Z" unit can be any unit you pick and will be shown for profiles or if you middle mouse press at any scan location to get a reading. And the units and it's conversions from DAC value are stored in NetCDF via the "dz" scale factor, the raw data field value is ALWAYS the original DAC value as of Gxsm philosophy, no matter unit or conversions. Simlpy for transparency.
Here I can imagine to add a separate conversion factor and custom unit for every channel.
-P
Last edit: Percy Zahl 2023-10-19
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Based on what you've said, it could indeed be useful to add a conversion factor and/or custom unit per channel within gxsm. I don't know how high of a priority it is, though; after all, no one has needed it thus far...
I think there is still value for gxsmread, because (i) people still have old saved data which may not have this configuration, and (ii) people may forget to set stuff up properly before running an experiment.
I'll still likely need to improve it in the future. It does not support spectroscopy files, and there may be some aspects to non-ADC channels that I need to consider (I haven't dealt with files that have these yet).
Thanks,
Nick
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Sure it is! I do have many scripts for various special data analysis tasks! Many even run right inside the Gxsm python console ;) And BTW, you can even swap units of scan data after the fact... but it's still a bit experimental -- using the build in python methods -- and sure if you wish, you do apply any scaling.
See also:
gxsm.set_scan_unit : Set Scan X,Y,Z,L Dim Unit: gxsm.set_scan_unit (ch,'X|Y|Z|L|T','UnitId string','Label string')
gxsm.set_scan_lookup : Set Scan Lookup for Dim: gxsm.set_scan_lookup (ch,'X|Y|L',start,end)
I'll have a look (Gxsm4 most likely only) if I can generalize the scan data unit and scale management better.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
In case it is useful for someone else, I've written a Python package to help read and parse gxsm scan files.
Why do this?
The huge flexibility of gxsm means that your ADC channels could correspond to anything. Since gxsm doesn't need to know the particulars of this (except perhaps for the nAmpere2Volt and dHertz2Volt parameters in the gxsm config), it does not request them. In turn, it saves the scanned data as ADC channels in ADC counter units. In short, it lets you interpret the data it saved, based on your SPM configuration.
This means you will likely be converting the data to physical units when interpreting them.
gxsm saves each scan channel in its own NetCDF3 file. It may be of interest to bundle all the channels of a given file together, for analysis purposes.
gxsmread is a package I wrote to help me do those 2 things. For (1), you can provide it a config file, indicating the conversion factor of each channel, units, and a name to give it; it will do the conversion for you. For (2), you can give it the path to a multifile dataset, and it will merge them all together.
None of this is particularly complicated, but not having to write it up each time may save others some time.
If interested, you can access it here: https://github.com/grutter-spm-group/gxsmread
Nick,
you brought up an very old -- going back to the roots of gxsm -- quest here also:
The pro and cons of storing channels in separated files! I am kind of glad you did. And wonder if any one else has some input here?
Facts:
* all "meta data" (minor from size for larger scans) besides the Z unit and 4D data set itself is a identical copy. Few exceptions are potentially attached user&probe data sets only been attached to the "master" scan, usually the 1st in line or Topo.
* File name convention, "channel name" and "Direction Mode" suffix to name.
* Any number scan channels, directions, etc.
* Easy channel data preview/selection via system file viewer via thumbnailing.
* Easy selective loading of particular only data channels for analysis.
* Easy data management or later removal of meaning less/obsolete data when forgotten to disable when monitoring additional channels at that time not used. Example left on dFreq/Exec data channels and went back to STM, or a like.
* Many files to manage... -- but on average 2..4 channels or 4x more files is not that big of an deal today.
any further points?
PS: I can -- very easy -- in future imagine for Gxsm4 to have the user choose and allow to store all channels withing one file. Anyways that will break the long back and forward compatibility of the Gxsm NetCDF format -- or will make additional channels inaccessible for older software! However, for Gxsm4 a move to the latest NetCDF4 (or5??) is in the to-do list anyways. It's only a tedious task to port from all the old netcdf lib code to the new... BTW -- nay takers to help?
Last edit: Percy Zahl 2023-10-20
Thanks for positing! Nice util.
PS: a few basic assignments are possible via preferences so. See screenshot.
It it helps, I could make custom per channel conversion factors to a core set of units available in future. For sure you can since ever "replace" the default names ADC1,2,... with any meaning full name there. And the file names will also have that post-fix name.
-P
Hi Percy,
Thanks for the quick reply! Of course, I am fairly new and did not realize there is this level of control in DataAq. I was dealing with old files that had presumably not been set up in this way before acquisition.
Out of curiosity: what does changing parameters in DataAq do? I assume changing the 'name' (e.g. DataAq/DataSrcA1) changes the saved filename, but what about Zunit and Zlabel? Of the files I have looked at (possibly outdated), all files have data stored in FloatField, and i don't recall there being units or labels.
Thank you,
Nick
Hi,
you must keep the "FloatField" type, as this is a hardware /HwI design requirement.
you can change name and label to anything, may avoid special characters. Like the default exanple shown for ADC0-ITunnel. (No need for ADC0, but I recommend to keep it and add info like "ADC3-TestSignal" so you later know where is was originated from, no need so!)
Label is used for the profile axis label (X or Y as used). Name is the long name is used also for the channel's file name post fix.
You can pick a unit from the lists, but the conversion available right now are limited to "length" Z (Ang/nm/...mm), Current (nA conversion) or Force (nN conversion) and the default "Volts".
"Z" (Z is historically at image pixel value's axis name) and the "Z" unit can be any unit you pick and will be shown for profiles or if you middle mouse press at any scan location to get a reading. And the units and it's conversions from DAC value are stored in NetCDF via the "dz" scale factor, the raw data field value is ALWAYS the original DAC value as of Gxsm philosophy, no matter unit or conversions. Simlpy for transparency.
Here I can imagine to add a separate conversion factor and custom unit for every channel.
-P
Last edit: Percy Zahl 2023-10-19
Hi Percy,
Based on what you've said, it could indeed be useful to add a conversion factor and/or custom unit per channel within gxsm. I don't know how high of a priority it is, though; after all, no one has needed it thus far...
I think there is still value for gxsmread, because (i) people still have old saved data which may not have this configuration, and (ii) people may forget to set stuff up properly before running an experiment.
I'll still likely need to improve it in the future. It does not support spectroscopy files, and there may be some aspects to non-ADC channels that I need to consider (I haven't dealt with files that have these yet).
Thanks,
Nick
Sure it is! I do have many scripts for various special data analysis tasks! Many even run right inside the Gxsm python console ;) And BTW, you can even swap units of scan data after the fact... but it's still a bit experimental -- using the build in python methods -- and sure if you wish, you do apply any scaling.
See also:
gxsm.set_scan_unit : Set Scan X,Y,Z,L Dim Unit: gxsm.set_scan_unit (ch,'X|Y|Z|L|T','UnitId string','Label string')
gxsm.set_scan_lookup : Set Scan Lookup for Dim: gxsm.set_scan_lookup (ch,'X|Y|L',start,end)
I'll have a look (Gxsm4 most likely only) if I can generalize the scan data unit and scale management better.