pymca-users

[PyMca] EDF file format

[PyMca g. p. m. list. <pym...@li...>]

Dear Armando,

We have written in our lab an acquisition software for our XRF system. I would like to directly write the data in EDF format but I can not find how to do it. Do you have a routine?

Thanks for your help,

David

David STRIVAY
Spectroscopie Atomique et Nucleaire, Archeometrie
Centre Europeen d’Archeometrie
U.R. Art, Archéologie et Patrimoine
IPNAS,Sart Tilman B15
Universite de Liege, 4000 Liege, Belgium
Tel: +32-4-3663695 - Fax:+32-4-3662884 - GSM: +32-479-451295

Re: [PyMca] EDF file format

[PyMca g. p. m. list. <pym...@li...>]

Dear David,

The simplest way to read and to write EDF files is to use the EdfFile.py
module. There are several flavours in the nature. You can use the one
shipped with PyMca or any other one:

https://github.com/vasole/pymca/blob/master/PyMca5/PyMcaIO/EdfFile.py

It is a stand alone file so it can be added to any project without
adding a dependency on PyMca.

Basically you just need to create the file:

from PyMca5.PyMcaIO import EdfFile
edf = EdfFile.EdfFile("yourfile.edf", access="ab+")
edf.WriteImage({}, your_numpy_array)
edf = None # to force to close the file

You should aim at generating one EDF per row of measured spectra.

If your raster experiment is n_rows x n_columns x n_channels you could
do something like:

import numpy
from PyMca5.PyMcaIO import EdfFile
data = numpy.array((n_columns, n_channels), dtype=numpy.float32)
# OPTIONAL: a, b and c are the coefficients of your calibration (c
expected to be 0.0)
ddict = {"Mca a": a,
              "Mca b": b,
              "Mca c": c}
for i in range(n_rows):
    for j in range(n_columns):
       data[j, :] = your_spectrum_data_for_pixel_i_j
    edf = EdfFile.EdfFile("root_name_%05d.edf" % i, access="wb")
    edf.WriteImage(ddic, data)
    edf = None
 
BTW, in what format are your original data? It would not be surprising
that they are already in a format supported by PyMca. Furthermore, if
that is not the case but you know how to read them from Python it would
cost almost nothing to add native support to them in PyMca.

However, I would strongly recommend you to use HDF5 if you are thinking
about saving stacks or even TIFF instead of EDF (PyMca can read
uncompressed TIFF files as if they would be EDF files).

I guess you are referring to the case where you collect N_CHANNELS
spectra on a regular grid of N_ROWS by N_COLUMNS. In its simplest form,
just a dataset with shape [N_ROWS, N_COLUMNS, N_CHANNELS] would do the
job.  That is the strict minimum. Your user should use latest PyMca
(v5.3.1 at this point) to be sure everything works properly.

You can make the life simpler to your user by adding some additional
conventions. The simpler you will make the life to your user, the harder
will be for you I recommend to add the attribute "interpretation" set to
"spectrum" to indicate that the dataset is a stack of spectra and not a
stack of images. The later would correspond to a stack of the form
[N_IMAGES, N_ROWS, N_COLUMNS] and it would be indicated by a an
attribute "interpretation" set to "image".  For XRF, it is recommended
the  [N_ROWS, N_COLUMNS, N_CHANNELS] arrangement.

If you encapsulate the dataset for each MCA device in a container group,
then you can add more information like the calibration, the live_time or
the elapsed_time of that MCA. For instance:

/whatever_name
/whatever_name/mca_0
/whatever_name/mca_0/data # dataset with shape [nrows, ncolumns,
nchannels] (regular grid) or [nspectra, nchannels] for non-regular grid
/whatever_name/mca_0/calibration # three values corresponding to a, b, c
in energy = a + b * channels + c * channels * channels ( with expected
to be 0 anyways)
/whatever_name/mca_0/channels  # dataset with the channel numbers. If
not supplied, it will be taken as 0, 1, 2, ...., nchannels-1
/whatever_name/mca_0/live_time # dataset if shape nrows * ncolumns with
the actual live time

I have started to improve the available documentation. It is a work in
progress, but if you go the HDF5 route, the link below might be of help:

 http://www.silx.org/doc/PyMca/dev/hdf5/index.html

Best regards, Armando

Best regards,

Armando

On 09/05/2018 10:49, PyMca general purpose mailing list. wrote:
> Dear Armando,
>
> We have written in our lab an acquisition software for our XRF system.
> I would like to directly write the data in EDF format but I can not
> find how to do it. Do you have a routine?
>
> Thanks for your help,
>
> David

Re: [PyMca] EDF file format

[PyMca g. p. m. list. <pym...@li...>]

Hi Armando and David,


My recommendation would be to go even one step further than what Armando suggests: write Nexus compliant HDF5 files-> http://www.nexusformat.org

Best regards,

Tom

> On 9 May 2018, at 10:23, PyMca general purpose mailing list. <pym...@li...> wrote:
> 
> Dear David,
> 
> The simplest way to read and to write EDF files is to use the EdfFile.py module. There are several flavours in the nature. You can use the one shipped with PyMca or any other one:
> 
> https://github.com/vasole/pymca/blob/master/PyMca5/PyMcaIO/EdfFile.py <https://github.com/vasole/pymca/blob/master/PyMca5/PyMcaIO/EdfFile.py>
> 
> It is a stand alone file so it can be added to any project without adding a dependency on PyMca. 
> 
> Basically you just need to create the file:
> 
> from PyMca5.PyMcaIO import EdfFile
> edf = EdfFile.EdfFile("yourfile.edf", access="ab+")
> edf.WriteImage({}, your_numpy_array)
> edf = None # to force to close the file
> 
> You should aim at generating one EDF per row of measured spectra.
> 
> If your raster experiment is n_rows x n_columns x n_channels you could do something like:
> 
> import numpy
> from PyMca5.PyMcaIO import EdfFile
> data = numpy.array((n_columns, n_channels), dtype=numpy.float32)
> # OPTIONAL: a, b and c are the coefficients of your calibration (c expected to be 0.0)
> ddict = {"Mca a": a,
>               "Mca b": b,
>               "Mca c": c}
> for i in range(n_rows):
>     for j in range(n_columns):
>        data[j, :] = your_spectrum_data_for_pixel_i_j
>     edf = EdfFile.EdfFile("root_name_%05d.edf" % i, access="wb") 
>     edf.WriteImage(ddic, data)
>     edf = None
>   
> BTW, in what format are your original data? It would not be surprising that they are already in a format supported by PyMca. Furthermore, if that is not the case but you know how to read them from Python it would cost almost nothing to add native support to them in PyMca.
> 
> However, I would strongly recommend you to use HDF5 if you are thinking about saving stacks or even TIFF instead of EDF (PyMca can read uncompressed TIFF files as if they would be EDF files).
> 
> I guess you are referring to the case where you collect N_CHANNELS spectra on a regular grid of N_ROWS by N_COLUMNS. In its simplest form, just a dataset with shape [N_ROWS,       N_COLUMNS, N_CHANNELS] would do the job.  That is the strict minimum. Your user should use latest PyMca (v5.3.1 at this point) to be sure everything works properly. 
> 
> You can make the life simpler to your user by adding some additional conventions. The simpler you will make the life to your user, the harder will be for you I recommend to add the attribute "interpretation" set to "spectrum" to indicate that the dataset is a stack of spectra and not a stack of images. The later would correspond to a stack of the form [N_IMAGES, N_ROWS, N_COLUMNS] and it would be indicated by a an attribute "interpretation" set to "image".  For XRF, it is recommended the  [N_ROWS, N_COLUMNS, N_CHANNELS] arrangement. 
> 
> If you encapsulate the dataset for each MCA device in a container group, then you can add more information like the calibration, the live_time or the elapsed_time of that MCA. For instance:
> 
> /whatever_name 
> /whatever_name/mca_0 
> /whatever_name/mca_0/data # dataset with shape [nrows, ncolumns, nchannels] (regular grid) or [nspectra, nchannels] for non-regular grid 
> /whatever_name/mca_0/calibration # three values corresponding to a, b, c in energy = a + b * channels + c * channels * channels ( with expected to be 0 anyways)
> /whatever_name/mca_0/channels  # dataset with the channel numbers. If not supplied, it will be taken as 0, 1, 2, ...., nchannels-1
> /whatever_name/mca_0/live_time # dataset if shape nrows * ncolumns with the actual live time
> 
> I have started to improve the available documentation. It is a work in progress, but if you go the HDF5 route, the link below might be of help:
> 
>  http://www.silx.org/doc/PyMca/dev/hdf5/index.html <http://www.silx.org/doc/PyMca/dev/hdf5/index.html> 
> 
> Best regards, Armando
> 
> Best regards,
> 
> Armando
> 
> On 09/05/2018 10:49, PyMca general purpose mailing list. wrote:
>> Dear Armando,
>> 
>> We have written in our lab an acquisition software for our XRF system. I would like to directly write the data in EDF format but I can not find how to do it. Do you have a routine?
>> 
>> Thanks for your help,
>> 
>> David
> 
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot_______________________________________________
> PyMca-users mailing list
> PyM...@li...
> https://lists.sourceforge.net/lists/listinfo/pymca-users

Re: [PyMca] EDF file format

[PyMca g. p. m. list. <pym...@li...>]

Hello,

If you are interested to take step suggested by Tom, the attached file
generates an HDF5 file with a stack and information that can be used by
PyMca (calibration, live_time,...)

Best regards,

Armando

On 09/05/2018 11:27, PyMca general purpose mailing list. wrote:
> Hi Armando and David,
>
>
> My recommendation would be to go even one step further than what
> Armando suggests: write Nexus compliant HDF5
> files-> http://www.nexusformat.org
>
> Best regards,
>
> Tom

[PyMca] different energy calibrations for multi-element detectors?

[PyMca g. p. m. list. <pym...@li...>]

Hi All: 

Greetings from CHESS. I have data obtained with a multi-element detector for which the gains are not precisely matched, and so each pixel requires a slightly different energy calibration. Does PyMCA have a way to deal with this — e.g. to apply different energy calibrations to each element?

Warm Regards,
-Arthur Woll

Re: [PyMca] different energy calibrations for multi-element detectors?

[PyMca g. p. m. list. <pym...@li...>]

Dear Arthur,

Thank you for a very nice question. Unfortunately the answer at this 
point is not yet. The good news is that the problem is in my TODO list 
because ESRF beamlines also need it.

The correct way to deal with the problem of several detectors would be 
to fit simultaneously all the detectors. The reason is that even if the 
calibrations can be forced to match, the geometry is not the same.

It is not particularly difficult to implement at least for the case of 
using the linear fit. I might come with a quick-and-dirty solution for 
this later case relatively soon.

Best regards,

Armando

On 13.07.2018 17:42, PyMca general purpose mailing list. wrote:
> Hi All:
> 
> Greetings from CHESS. I have data obtained with a multi-element
> detector for which the gains are not precisely matched, and so each
> pixel requires a slightly different energy calibration. Does PyMCA
> have a way to deal with this — e.g. to apply different energy
> calibrations to each element?
> 
> Warm Regards,
> -Arthur Woll
> 

Re: [PyMca] different energy calibrations for multi-element detectors?

[PyMca g. p. m. list. <pym...@li...>]

Hi Armando,

Thanks for the reply!  I understand what you mean, but what I suppose I had in mind was that you could have linear corrections to the detector cals applied independent of the E-cal parameters that PyMCA applies during fitting. What follows may not be practical to implement, but I’d imagined is that if you first find Ecals for each of four detectors separately: 

E_i = A_i + B_i*ch_i

you could then determine correction factors dA_i, dB_i, that would account for only the differences among the detectors. (e.g. define dA_1 = 0, then dA_2 = A_2 - A_1, dA_3 = A_3 - A_1, etc) These would then be fixed, but could be used in addition to “master” coefficients that would be the same for each detector, and would be treated the way you normally treat coefficients. (and so could still be allowed to vary for a non-linear fit):

E_i = A + B*ch_i + (dA_i + dB_i*ch_i)

But it’s hard to see how this wouldn’t be a bit cumbersome on the input side...

-Arthur


> On Jul 14, 2018, at 1:54 AM, PyMca general purpose mailing list. <pym...@li...> wrote:
> 
> Dear Arthur,
> 
> Thank you for a very nice question. Unfortunately the answer at this point is not yet. The good news is that the problem is in my TODO list because ESRF beamlines also need it.
> 
> The correct way to deal with the problem of several detectors would be to fit simultaneously all the detectors. The reason is that even if the calibrations can be forced to match, the geometry is not the same.
> 
> It is not particularly difficult to implement at least for the case of using the linear fit. I might come with a quick-and-dirty solution for this later case relatively soon.
> 
> Best regards,
> 
> Armando
> 
> On 13.07.2018 17:42, PyMca general purpose mailing list. wrote:
>> Hi All:
>> Greetings from CHESS. I have data obtained with a multi-element
>> detector for which the gains are not precisely matched, and so each
>> pixel requires a slightly different energy calibration. Does PyMCA
>> have a way to deal with this — e.g. to apply different energy
>> calibrations to each element?
>> Warm Regards,
>> -Arthur Woll
> 
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> PyMca-users mailing list
> PyM...@li...
> https://lists.sourceforge.net/lists/listinfo/pymca-users