pymca-users

[PyMca] Documentation for FastXRFLinearFit

[PyMca g. p. m. list. <pym...@li...>]

Dear Armando,

I was wondering if you could point me toward the documentation for the fast
XRF linear fitting plugin? I would like to know what is the underlying
algorithm and how do the weightings work? Is the background also
subtracted? Etc.?

Thanks a million!

Best,
Marc

Re: [PyMca] Documentation for FastXRFLinearFit

[PyMca g. p. m. list. <pym...@li...>]

Dear Marc,

What follows is a copy-paste of the text from the slides I use for training:

"""
As you may have found out with the standard fit, maximum speeds can be
achieved when you can afford a linear fit:

- Non linear parameters have been previously determined and kept fixed
- No pile up is considered (remember: pile up is not the same on each
pixel and average pile up has no meaning

On a linear fit, the signal can be written as a matrix product:

Measured Signal B = Design Matrix * A

In other words, the measured spectrum B is obtained as a linear
combination of the columns of the Design Matrix and we have to find the
coefficients A on that basis.

The uncertainties on the measured data enter as weights in the Design
Matrix that has n_channels rows and m_model_parameters columns.

Therefore, in the linear fit we need the "equivalent of a matrix
inversion and a matrix multiplication" to obtain the A parameters.

If we assume that the uncertainty associated to the counts in a channel
is the same on all the measured spectra, the Design Matrix will be the
same in every point and only one matrix inversion equivalent is needed.

That corresponds to:

- Using No Weight (assume uncertainty is 1 everywhere) or
- Use an average Weight (derive channel uncertainty from the sum spectrum)

All that is achievable from the GUI or from the command line (no GUI) 
in source installation of PyMca:

python -m PyMca5.PyMcaPhysics.xrf.FastXRFLinearFit
--cfg=configuration_file --weight=0 --tif=1 --concentrations=0 --refit=0
hdf5_file_name::dataset_path

"""

Concerning your specific question: The background is accounted for or
not following what you requested via the fit configuration file.
Remember that all the non-linear parameters will be fixed to their
values in the fit configuration file. If they are wrong they will remain
wrong. To help on that, the fit configuration window allows you to load
the parameters of the last fit into the configuration ("Load from Fit").

Best regards,

Armando

Re: [PyMca] Documentation for FastXRFLinearFit

[PyMca g. p. m. list. <pym...@li...>]

Dear Armando,

This is a very useful explanation. As I now understand the algorithm, you
are finding the least-squares solution to the classic Ax=b problem. This
also seems to be what Chris Ryan and the Geopixie software suite is doing
with their 'Dynamic Analysis'. I am, however, still confused about how the
uncertainty weights are applied to the 'Design Matrix'. Is there a separate
matrix of weights? How are these weights predetermined? Is their physical
basis really counting statistics as you suggest?

It would be useful if you could provide a little more mathematical
description of this aspect of the Fast Linear XRF fit algorithm.

Thanks! And Happy Holidays!

Best,
Marc

On Fri, Dec 6, 2019 at 10:58 AM PyMca general purpose mailing list. <
pym...@li...> wrote:

> Dear Marc,
>
> What follows is a copy-paste of the text from the slides I use for
> training:
>
> """
> As you may have found out with the standard fit, maximum speeds can be
> achieved when you can afford a linear fit:
>
> - Non linear parameters have been previously determined and kept fixed
> - No pile up is considered (remember: pile up is not the same on each
> pixel and average pile up has no meaning
>
> On a linear fit, the signal can be written as a matrix product:
>
> Measured Signal B = Design Matrix * A
>
> In other words, the measured spectrum B is obtained as a linear
> combination of the columns of the Design Matrix and we have to find the
> coefficients A on that basis.
>
> The uncertainties on the measured data enter as weights in the Design
> Matrix that has n_channels rows and m_model_parameters columns.
>
> Therefore, in the linear fit we need the "equivalent of a matrix inversion
> and a matrix multiplication" to obtain the A parameters.
>
> If we assume that the uncertainty associated to the counts in a channel is
> the same on all the measured spectra, the Design Matrix will be the same in
> every point and only one matrix inversion equivalent is needed.
>
> That corresponds to:
>
> - Using No Weight (assume uncertainty is 1 everywhere) or
> - Use an average Weight (derive channel uncertainty from the sum spectrum)
>
> All that is achievable from the GUI or from the command line (no GUI)  in
> source installation of PyMca:
>
> python -m PyMca5.PyMcaPhysics.xrf.FastXRFLinearFit
> --cfg=configuration_file --weight=0 --tif=1 --concentrations=0 --refit=0
> hdf5_file_name::dataset_path
>
> """
>
> Concerning your specific question: The background is accounted for or not
> following what you requested via the fit configuration file. Remember that
> all the non-linear parameters will be fixed to their values in the fit
> configuration file. If they are wrong they will remain wrong. To help on
> that, the fit configuration window allows you to load the parameters of the
> last fit into the configuration ("Load from Fit").
>
> Best regards,
>
> Armando
> _______________________________________________
> PyMca-users mailing list
> PyM...@li...
> https://lists.sourceforge.net/lists/listinfo/pymca-users
>

Re: [PyMca] Documentation for FastXRFLinearFit

[PyMca g. p. m. list. <pym...@li...>]

Dear Marc,

On 11/12/2019 15:36, PyMca general purpose mailing list. wrote:
> Dear Armando,
>
> This is a very useful explanation. As I now understand the algorithm,
> you are finding the least-squares solution to the classic Ax=b problem.

Yes, the whole point is that when the A matrix is the same for all the
spectra, you only need one matrix spectrum.


> This also seems to be what Chris Ryan and the Geopixie software suite
> is doing with their 'Dynamic Analysis'. I am, however, still confused
> about how the uncertainty weights are applied to the 'Design Matrix'.
> Is there a separate matrix of weights?

An image will help you most.

Take a look at Figure 15.4.1 in Numerical Recipes 2nd or 3rd edition or
at Figure 14.3.1 in Numerical Recipes 1st edition. Chapter Modeling of
data, section General Least Squares. If you do not have the book, you
can google for "numerical recipes pdf" you should find it.

> How are these weights predetermined? Is their physical basis really
> counting statistics as you suggest?

Yes. The uncertainty on each point is taken either 1 (no weight),
sqrt(sum_spectrum)/nspectra (average) or sqrt(ncounts). If you are going
to use the later, it is better that you use the standard batch in PyMca
(it will be faster).

As you see in the figure, the uncertainties on the data points enter in
the A matrix.

>
> It would be useful if you could provide a little more mathematical
> description of this aspect of the Fast Linear XRF fit algorithm.
>
Really, it is just a standard weighted linear fit. Please read the
chapter mentioned above.

Armando

Re: [PyMca] Documentation for FastXRFLinearFit

[PyMca g. p. m. list. <pym...@li...>]

On 11/12/2019 16:37, PyMca general purpose mailing list. wrote:
> Dear Marc,
>
> On 11/12/2019 15:36, PyMca general purpose mailing list. wrote:
>> Dear Armando,
>>
>> This is a very useful explanation. As I now understand the algorithm,
>> you are finding the least-squares solution to the classic Ax=b problem.
>
> Yes, the whole point is that when the A matrix is the same for all the
> spectra, you only need one matrix spectrum
>
Sorry, it should be written one "matrix inversion".

Armando

Re: [PyMca] Documentation for FastXRFLinearFit

[PyMca g. p. m. list. <pym...@li...>]

Thank you so much Armando. This is now crystal clear to me.

Best,
Marc


On Wed, Dec 11, 2019 at 9:38 AM PyMca general purpose mailing list. <
pym...@li...> wrote:

> Dear Marc,
>
> On 11/12/2019 15:36, PyMca general purpose mailing list. wrote:
>
> Dear Armando,
>
> This is a very useful explanation. As I now understand the algorithm, you
> are finding the least-squares solution to the classic Ax=b problem.
>
> Yes, the whole point is that when the A matrix is the same for all the
> spectra, you only need one matrix spectrum.
>
>
> This also seems to be what Chris Ryan and the Geopixie software suite is
> doing with their 'Dynamic Analysis'. I am, however, still confused about
> how the uncertainty weights are applied to the 'Design Matrix'. Is there a
> separate matrix of weights?
>
> An image will help you most.
>
> Take a look at Figure 15.4.1 in Numerical Recipes 2nd or 3rd edition or at
> Figure 14.3.1 in Numerical Recipes 1st edition. Chapter Modeling of data,
> section General Least Squares. If you do not have the book, you can google
> for "numerical recipes pdf" you should find it.
>
> How are these weights predetermined? Is their physical basis really
> counting statistics as you suggest?
>
> Yes. The uncertainty on each point is taken either 1 (no weight),
> sqrt(sum_spectrum)/nspectra (average) or sqrt(ncounts). If you are going to
> use the later, it is better that you use the standard batch in PyMca (it
> will be faster).
>
> As you see in the figure, the uncertainties on the data points enter in
> the A matrix.
>
>
> It would be useful if you could provide a little more mathematical
> description of this aspect of the Fast Linear XRF fit algorithm.
>
> Really, it is just a standard weighted linear fit. Please read the chapter
> mentioned above.
>
> Armando
> _______________________________________________
> PyMca-users mailing list
> PyM...@li...
> https://lists.sourceforge.net/lists/listinfo/pymca-users
>