#354 Hooks by default in scans

[Teresa Nunez]

Hi,
I copy here an e-mail that I sent some time ago, it has
become now a request from our users.

our beamline scientists want that all the scans run by default
with hooks functions, not like it is now that it is necessary to define
a new macro running an existing scan macro after adding hooks to it.
They want to keep the original names for the current scan macros and
simply define a function that will be run always like a hook.
I think that the current implementation for the hooks works quite
well, and offers flexibility for example for the arguments that have
to be passed. A fixed function could have some problem there.
So in my opinion one should keep what is implemented and try to
add the requested feature that could be activated by an additional
variable (set for example in the spock environment).
What do you think about something like this?.

                        Regards,
                                   Teresa

[Teresa Nunez]

Hi again,
only for avoiding parallel work in the same subject ...
I have implemented a first approach for getting this done. The user
can simply define some functions (with fixed names) and they will be
run during the scan (like the hooks: pre-move, pre-acq, ...), even
if these functions are of course not macros, one can have the macro
available inside, so one can profit from all the advantages that this
implies. It will need for sure some cleanup and some more details ... but
perhaps it could be used as start point.

                   Regards,
                               Teresa

[Carlos Pascual]

looks nice.
Just one question:
once a hook function is defined (e.g. pre-move) how do you control which macro uses it and which doesn't? (imagine I want ascan to use it but I do not want mesh using it)

One possible way would be to use environment variables (which already allow for several levels of scopes). And as a bonus, you do not need to hardcode the names of the functions, because the value of the variable could be a dictionary mapping hook names with function and/or macro names.

For example:

# Assume m1, m2 are macros defined somewhere in the macropath
# Also assume that myhooks.py is a module located 
# somewhere in the pythonpath and which defines
# functions called f1, f2, ...

# set the macros m1 and m2 as the pre-move hooks for ascan only
senv ascan.HOOKS "{'pre-move':['m1', 'm2']}"

# set f1 as the post-acq hook for all macros that use post-acq hooks
senv HOOKS "{'post-acq':['myhooks.f1']}"

# set f2 as the pre-move hook for ascan only
senv ascan.HOOKS "{'pre-move':['myhooks.f2']}"

# set both f3 and f4 as the pre-ack hooks for all macros on MyDoor
senv MyDoor.HOOKS "{'pre-move':['myhooks.f3','myhooks.f4']}"

A further sophistication would be to accept args and kwargs using an optional length-3 tuple notation such as:

# assume f5 accepts two args plus the x and y kwargs:
senv HOOKS "{'post-acq':[('myhooks.f5', (1,2), {x=3,y=4})]}"

# assume we want to always move mymotor to 0 before all scans
senv HOOKS "{'pre-scan':'[('mv', ('mymotor 0'), {})]}"

[Teresa Nunez]

Hi Carlos,
many thanks for your comments and suggestions.
I didn't want to read much more from the environment, but
I will test it, because I think it is really a good idea
to make it like this. Many thanks.

                   Regards,
                              Teresa

[marc]

Hi Teresa,

We would like to have a look to the DESY solution to the hooks in order to be able to discuss about this subject during the Sardana Workshop. Could you please push your implementation to a remote branch where we could have a look to it?

We would like to ask you if you would like to present the 'hooks' during workshop at DESY. In this case it would be nice to present both ideas, the DESY implementation and the Carlos idea about using Sardana environment variables. What do you think?

Regards,
marc

[Teresa Nunez]

Hi Marc,
I will try to put the implementation in some branch, I have
first to make some changes in my files ... I mean, go back
to the versions in the sardana develop branch (we have some
changes at DESY), but I hope I will manage in the next days.
Of course, I can present something about the hooks, in fact there
is already a point in the Workshop program about that.

                       Regards,
                                          Teresa

[Teresa Nunez]

Hi Marc,
for the implementation of the general hooks I had
modified only one file (gscan.py). I have just now taken the current
version of this file from the sardana develop branch and applied
the changes. I send you the file per e-mail with documentation
how to used (I tried to put the link here, but it doesn't work).
Tell me if you have problems trying it. Many thanks.

                   Regards,
                                        Teresa

[Teresa Nunez]

The file with the implementation

[Teresa Nunez]

Documentation how to use it:

http://hasyweb.desy.de/services/computing/Spock/

Section:

Scans -> Hooks -> General hooks, conditions and on_stop function

[Roberto Javier Homs Puron]

After analyzing the Teresa's solution, IMHO it is not generic. It uses a specific file that it should be on the PYTHONPATH of the MacroServer PC and the order of the hook execution could be important. I agree with the Carlos's solution (to use the environment), because it is independent of any file, but the current way to edit the environment is not user-friendly (too complicated).

In the meantime, we can find an alternative solution to set the hook in the environment, e.g: with a specific GUI (similar to expconf) or with specific macros. Another solution is to set an environment variable to specify the file with an API (to be develop) to configure the hooks. If we want a complex solution but generic, we can create a HookManager which is responsible to manage the different entry points (file, function, environment, etc.).

[Teresa Nunez]

Hi Roberto,
many thanks for your comments and sorry for not answering
until now, I did not have any time for looking at it again.
Here some replies to your comments:

'It uses a specific file that it should be on the PYTHONPATH of the MacroServer PC'

It is not in the PYTHONPATH, we add the path to the PythonPath property of
the MacroServer device. About the MacroServer PC, well, this is the case for
all the files used by the MacroServer, included Macros or any external module.

'the order of the hook execution could be important'

I don't know what do you mean with that. The order is und has to be important.

' I agree with the Carlos's solution (to use the environment), because it is independent of any file, but the current way to edit the environment is not user-friendly (too complicated).'

We can handle this with macros, I think this is not the problem, the main problem for
us was the overloading of the environment and that the users do not like the idea
of using it. But well, it can be done with macros and keep transparent for the users
that everything is in the environment ... even if they will see it ... I don't think that
GUIs are a good idea, for being used at DESY, I mean. But how to set the environment
is independent of the implementation, every one could use like it is prefered.
Should I try to change the current implementation following the hints from Carlos or
do you prefer to discuss before any more possible scenarios, implementations ...?.

                     Regards,
                                             Teresa

[Zbigniew Reszela]

Teresa, I think that the solution with the environment is more flexible and more standard. It would be great if you could modify your solution and do it this way.

I think that we should agree what to do in the case where macro has explicitly defined hooks in its code. I see the following options:

• just execute the hooks defined in the macro, and ignore the ones from the environment
• just execute the hooks defined in the environment, and ingore the ones from the macro
• execute all hooks, first the ones from the macro and then the ones from the environmnet
• execute all hooks, first the ones from the environmnet and then the ones from the macro
• abort the macro and warn about the conflict

Personally I like the first option.

Regarding the environement variable name, I would use the CamelCase writing and call it Hooks. What do you think?

[Teresa Nunez]

Hi Zibi,
many thanks for your commets.
I will work on the change of the implementation.
Abot the options you list, here we use the solution 4:

execute all hooks, first the ones from the environmnet and then the ones from the macro

For us the discussion was between the third and the fourth
you list, we want our general hooks to be executed always, but
in addition we can add specific hooks in the own code of a given
macro. We can not work with the first solution. We have macros
with own hooks defined, they users could not know about that,
and they want in any case that the general hook will be always
applied, also to these macros. If you don't agree with this option,
we can make it selectable.

                  Regards,
                                          Teresa

[Zbigniew Reszela]

Initially I was thinking that the explicit definition of the hooks in the code should be the only one executed. But as you say, from the user point of view it will be hard to know about the implementation details. I agree with you, but let me ask here at Alba... We will reply in this thread if there are some arguments agains option 4.

[Teresa Nunez]

Hi,
I have just sent a patch to the mail list with an implementation of the
general hooks based on a dictionary enviroment variable called GeneralHooks.
The implementation allows the use of macros, with any argument also given in
the same environment variable, as proposed by Carlos, as general hooks.
We have decided that this is enough and even better for us. Our general functions
were passing always the macro as argument, because some features of the macros,
like execMacro or the spock output were usually needed, so it is better that they
are already macros. Please have a look to the implementation and send your comments.
The environment variable can be set directly from spock like:

senv GeneralHooks {\"pre-scan\":[(\"mv\" \"exp_dmy01\" 50)] \"pre-move\":[(\"mv\" \"exp_dmy02\" 50)]}

But I will make macros for setting everything via Macros, so that the users do not have
to take care of this complicated syntax. We will also probably fix the name of the macros
for being run as general hook at each point in our spock environment, making everything
even easier for the users, but this is something that has nothing to do with the
implementation.
I have not done anything for the continuous scans, since they are changing considerable
in the SEP6. But it also could be extended if needed.
I will also attach the patch to this ticket.

                                  Regards,
                                                         Teresa

[Teresa Nunez]

Here the patch.

[Teresa Nunez]

Hi,
I have also changed the implementation of the general condition for checking
if a point of the scan should be repeated in the same way, using another environment
variable that should be an string with the macro to be run for checking (also arguments
can be given). I do not send the patch, because in any case for the discussion is enough
to see the implementation of the general hooks.

                          Regards,
                                           Teresa

[Teresa Nunez]

Hi,
in the same way I have also changed the implementation for the general
stop function. The difference with the two implementations above is that in this
case it has to be a function (a macro is not posible because with ctrl-C we stop
all macros, so call to macro members like execMacro does not work any more).
I think for the general hooks and condition we can keep only macros, but if you
want to extend it to functions, it could also be done. In my opinion is not needed,
and I say in some previous mail, the macro was even always sent as an argument
to the function because it was need it.

                       Regards,
                                            Teresa

[Teresa Nunez]

Hi,
I would like to know your opinion about one point. What should happen if
an execution of a general hook macro fails or if the macro is not found?, and
the same question for the general condition and stop function. In the patch
I have sent the scan will be stopped if any general hook fails. But I think it
would be better if the scan ignores this and it continues. For the general stop,
for example, I already implemented ignoring the error ang giving a warning.
I will change the implementation of the general hooks and conditions in the
same way. In the general condition, if the check of the condition fails the point
will not be repeated and a warning will be printed out. Do you agree with this?.

               Regards,
                                    Teresa

[Carlos Pascual]

Hi Teresa, I was about to integrate this feature, but I realized that you mention some other changes that are not sent.

Could you please send all the patches so that I integrate it all?

[Teresa Nunez]

Hi Carlos,
I send a patch with the last implementation.
I have also made some macros to set the environment, they have still with a preliminary name to not conflict with the old implentation I did, which is still used in some beamlines.
I send also the file with the macros and some info, this helps to know how to use the implementation:

• For general hooks:

-- new_gh_enable -> enable hooks, if not argument is given enable all hooks with
default values:
'pre-scan': ['gh_pre_scan'],
'pre-move': ['gh_pre_move'],
'pre-acq': ['gh_pre_acq'],
'post-acq': ['gh_post_acq'],
'post-move': ['gh_post_move'],
'post-step': ['gh_post_step'],
'post-scan': ['gh_post_scan']

For giving other names, ex. for a post-scan hook:

new_gh_enable "mymacroforpostscan oneargument otherargument" post-scan

or several macros:

new_gh_enable "mymacroforpostscan arg1 arg2, myseconmacro arg3" post-scan

-- new_gh_disable -> disable all general hooks

-- new_gh_isEnabled -> return true and macro names if general hooks are enabled.

• For general conditions:

-- new_gc_enable -> enable general conditions, if not argument is given the default name

                     gc_macro

is used, so a macro called like this will be called for checking the condition.
For setting any other macro, ex.

new_gc_enable macro_withoutargs

new_gc_enable "macro_withargs arg1 arg2"

Only one macro is allowed, since a condition is checked and it is better to have all the logic in
one macro and not start making 'and'/'or' conditions betweenn different macros.

-- new_gc_disable -> disable general condition

-- new_gc_isEnabled -> return true and name of the macro if the general condition feature is
enabled.

• For general stop:

-- new_gs_enable -> enable general stop feature, if not argument is given a function called
general_on_stop in the module general_functions will be called. An argument will be the name
of the function and the module it belong to, ex:

new_gs_enable mymodulename.new_onstop_function

for calling the function new_onstop_function defined in the file mymodulename.py. The file has to
be in a directory included in the macropath property of the MacroServer device.

Also only one function is allowed. If you want to switch between several according to the hook you
are running, call in your hooks this macro for setting the on stop function to the corresponding name.

-- new_gs_disable -> disable general stop feature

-- new_gs_isEnabled -> return true and name of the function if general stop feature is enabled.

• All features together:

new_gf_status -> display the status of the general features: hooks, conditions, on_stop
new_gf_enable -> enable all general features to default names
new_gf_disable -> disable all general features

The 'new' in the name of the macros will disapear, I only keep it like this for the moment for
not having conflicts with the previous implementation.

                                       Regards,
                                                              Teresa

[Teresa Nunez]

The patch

[Carlos Pascual]

Hi,
I haven't finished the review yet (I just reviewed the code, but didn't do any tests). But there are some comment already:

• In macro.py you define getGeneralHooksC method which looks identical to getGeneralHooks and not called, AFAIK. Maybe it is some forgotten garbage from develpment? UPDATE: I just realized that this is for cscan... It would be nice to have a single mechanism...
• For simplicity for the user, I would prefer a similar API for all cases (hooks, stop and conditions). This means supporting functions for conditions and hooks. It also means defining some syntax to differentiate a string meant to be executed as a macro from a string meant to be evaluated with eval().
• in the getGeneralOnStopFunction I would avoid "completing" the envvar contents with parenthesis (i.e. I would force the user to always write parenthesis explicitly). This is for the following reasons:
  • it saves us from the ugly (non-robust) usage of "find".
  • it would a allow for a natural way of differentiating macros from functions (see previous point). In this case, I would implement the differentiation using a regexp (which makes it more robust than using find/strip/etc)
• I do not like the use of "gs_module" as a special name. I would just make sure that whatever needed modules are (re)loaded, and I would avoid the call to .replace(self.module_to_import, "gs_module"). If for some reason it is really needed to implement it like this, I would at least call the module _gs_replace to indicate that it is a private thing

I am aware that some of my proposed changes may break your existing workflow, so just take them as a proposal to find a solution that is ok for all.

[Teresa Nunez]

Hi Carlos,
many thanks. I add hier the answers again.
Point 1. You have already found it.
Point 2. I did not put the functions because it does complicate the implementation
without gain. I think it is cleaner and easier to use only with macros. Our users prefer
now this implementation, and we also prefer to give them a way to do it without too
much thinking between different possiblities. In the previous implementations we had
only functions and it was much more confussing to use.
Point 3. No problem, I put it like this again for simplicity (in this case for the users).
The general on_stop can not be a macro, but well, if we want to put the hooks and
conditions also as functions, will be a way of differentiating.
Point 4. If you suggest another way, no problem, it was the only way I found for being
sure that the modules are always reloaded when the functions are called, without the
users having to take care of that.

                                  Regards,
                                                         Teresa

[Carlos Pascual]

Some more comments.

In your patch you create a mechanism for executing the "general hooks" which is parallel to that of the normal hooks.
Instead, I would prefer to extend the original mechanism to support the definition of hooks via the envvar. (i.e. merging macro.getGeneralHooks and macro.getHooks, and then also extend the part of gscan where the hooks are executed)

Also, for API simplicity, the onStopFunction could be implemented as just another hook. AFAIK, the stop function is similar to the "post-scan" hint with maybe the difference that it is executed no matter what (i.e. it would be executed in the "finally" clause of a try-except-finally construction... in fact maybe it could just be a hook with "hint='finally'" )

[Teresa Nunez]

Hi Carlos,
we want to have the two different implementations, the hooks
called from the macro in which you have defined them, and the hooks that
are general and then affecting all the macros. Do you mean to define only one
function but making what now make getGeneralHooks and getHooks exactly
like it is done now?.
I don't see your last point, sorry. The use of the onStopFunction should not imply
any extra change in any macro, adding any kind of hint or function to it. It has
to be only a function that will be executed any time and only when a ctrl-c is
done. I have tried to keep the current implementation without the minimum
modifications of Sardana (mainly because we are already using it and I have to
keep the changes every time we take again the develop branch from Sardana).
If you prefer to make a major change, no problem.
Again thanks for the review.

                         Regards,
                                              Teresa