#445 allow easy scan data analysis in order to take action based on the result e.g. goto_peak

[Zbigniew Reszela]

The following thread in the devel mailing list could serve as an introduction to this request.

One of the examples where this could be applied is the diffractometer alignment process. It consists of scans of the diffractometer geometry angles and looking for the intensity peak on one of the experimental channels. Subsecuently users are interested in sending the scanned motor to the position corresponding to the peak.
So this is usually a two step process requiring user acknowledge of the final move.

At the moment this could be achieved in several ways:

• Pure macro solution - develop a wrapping macro that would:
  1. Execute a standard scan.
  2. Access to the scan data using the scan macro object and analyze it
  3. Eventually ask the user if he would like to go to the peak position (input macro)
  4. Execute the move.
• Macro + external process solution:
  1. Execute a standard scan.
  2. Access the scan data using the RecordData Door's attribute from another process e.g. GUI (this applies only to the last executed macro) and analyze it.
  3. Inform the user about result of the analysis
  4. Execute the move.

This feature-request is to answer if the present solutions are generic enough to cover all the needs and eventually design and implement better solutions.

We could also check how this is solved in SPEC program.

[david fernandez]

Dear All,
In my opinion, here applies Voltaire's "the better is the enemy of the good". I think this to be managed by the experiment configuration expconf. The way I would do this is defining there the channel on which statistics will be applied. This is the way spec does (...used to do) and I think it is not "generic" but a good compromise. In this way statistics would only be applied in one channel (peak, max, min, fwhm, ...). It could even be printed at the end of the scan and/or displayed in the plot. This would be also the reference for the macros peak (go to peak) after the scan. Data could be accessed either way (proposed by Zibi), although, in order to limit very many very short access to the file (in disks often optimized to stream in large files and imanges but not for short access), I would keep if possibe the scan data (scalars) in memory.... (IMHO:)...brainstorming). A macro could also easily access other arbitrary channels (all of them) to calculate statistics on a particular scan when needed...

[Carlos Pascual]

I do not like "the SPEC" approach. In my opinion, a key design point is to separate the "statistics calculation" from the scan itself. The reason is that there is no one-size-fits-all statistics solution. The array of data resulting from a scan may be of any arbitrary form (a peak, a multi-peak, a noisy exponential decay,... whatever). Therefore it makes no sense to try to calculate domain-specific statistics for such a general case (e.g., what is the meaning of a FWHM for a noisy but linear function?).

The statistics should be something you can choose to do after the scan, and it should be modular (i.e. one should be able to choose which statistics to perform... e.g., in the linear function case it may be way more interesting to find the "slope" than the FWHM).

Therefore, the mentioned "scan-and-go-to-peak" use case should be a particular case implemented with 3 independent macros:

{scan} --> {calculate_peak(scandata)} --> {move(peak_position)}

So, IMHO, the design should focus on how to deliver the scan data from the scan macro to the particular statistics macro(s) and then how to transfer the calculated data from the statistics macro(s) to the "action" macro.
If one uses a wrapper macro, this is already possible (and in fact it is already being done in some user-specific macros). The challenge is to provide an interface to do this without having to program specific macros.

I think that there is a generic solution that can be used in both cases: we just need to assign an ID to the "output data" of every macro we have executed, and keep a pointer to where such data is actually available (disk or memory or a tango attribute,... whatever).

Then we need to provide a mechanism to access the data by ID (something like Ipython's usage of _N to access its Nth output). The ID could be as simple as a generalization of the scanID to all macros, which would allow us to use negative numbers to refer to relative macros (e.g. "-1" would refer to the last executed macro).

Regarding the actual storage place: for non-large scans as well as for the last statistics macros, we can keep the data in a cache in memory (an envvar or a tango attr). For the rest, we can just retrieve data from disk (this is similar to what the showscan macro is currently doing).

One final thing: for the internal implementation of data reference and retrieval, it could all be based on taurus attribute URIs if we implement both the "h5file" and "msenv" taurus schemes.

[Teresa Nunez]

Hi Carlos,
the aproach you mention is also like we make it at DESY in
some private implementations. I also prefere this way.

                     Regards,
                                        Teresa