Re: [Lmod-users] [EXTERNAL] Re: changes to tcl 'break' statement in lmod 8 series
A Lua based environment module system that reads TCL modulefiles.
Brought to you by:
rtmclay
Menu
▾
▴
Re: [Lmod-users] [EXTERNAL] Re: changes to tcl 'break' statement in lmod 8 series
|
From: Shrader, D. L. <dsh...@la...> - 2022-04-15 14:10:29
|
Hey Robert,
We just tried 8.6.18, and our problematic modulefile that uses a break statement inside a for loop works as expected. Thank you so much for the fix!
Thanks again,
David
________________________________
From: Robert McLay <mc...@ta...>
Sent: Thursday, April 14, 2022 2:16 PM
To: Shrader, David Lee
Cc: lmod-users
Subject: Re: [EXTERNAL] Re: [Lmod-users] changes to tcl 'break' statement in lmod 8 series
Well, I was able to track down how break works in TCL. A break inside a loop exits the loop. A break outside the loop causes an error. This error can be caught and cancel the evaluation of the current module. I have run TCL modules with breaks inside a loop as well as outside a loop and it now works for me. Please test Lmod 8.6.18 to see if it works for you.
Best,
Robert.
On Thu, Apr 14, 2022 at 10:17 AM Shrader, David Lee <dsh...@la...<mailto:dsh...@la...>> wrote:
What might be very irritating to deal with would be if a break/stop-loading-this-modulefile happens inside a for loop. Maybe a different function name would have to be used other that "break"?
Thanks again for looking in to it.
David
________________________________
From: Robert McLay <mc...@ta...<mailto:mc...@ta...>>
Sent: Wednesday, April 13, 2022 5:29 PM
To: Shrader, David Lee
Cc: lmod-users
Subject: Re: [EXTERNAL] Re: [Lmod-users] changes to tcl 'break' statement in lmod 8 series
Break is more tricky then I thought it was. I'll have to figure out how break works in TCL and tmod. This is going to take some time. "break" will break out of a loop. But if it is a bare "break" it does "I'm not loading the current module" thing and I don't know how to do that in TCL yet.
Best,
Robert.
On Wed, Apr 13, 2022 at 5:54 PM Shrader, David Lee <dsh...@la...<mailto:dsh...@la...>> wrote:
Hey Robert,
I have a follow-up scenario to the issue of "break" in tcl modulefiles with Lmod 8.6+. With the new way of dealing with "break", it is now not possible to break out of a for loop in tcl. Whenever this is used, Lmod just exits, on module load and unload. We've got at least one modulefile that does this at our site, and there may be more (we haven't put the newest Lmod on systems with many more user modulefiles on it).
Would a possible solution to this be to just use `--with-fastTCLInterp=no`, as came up earlier in this email chain?
Thanks,
David
________________________________
From: Shrader, David Lee
Sent: Tuesday, March 22, 2022 4:25 PM
To: Robert McLay
Cc: lmod-users
Subject: Re: [EXTERNAL] Re: [Lmod-users] changes to tcl 'break' statement in lmod 8 series
That is a good point. Again, thank you for all of the help!
David
________________________________
From: Robert McLay <mc...@ta...<mailto:mc...@ta...>>
Sent: Tuesday, March 22, 2022 3:43 PM
To: Shrader, David Lee
Cc: lmod-users
Subject: Re: [EXTERNAL] Re: [Lmod-users] changes to tcl 'break' statement in lmod 8 series
You would still get the "error" message in the case where the module reports the message. It is just that the "error" becomes a warning. This means that the warning is reported and the evaluation of the modulefile continues. The user requested the module be unloaded. The user got what they wanted. This is the best I can do in a bad situation.
Best,
Robert.
On Tue, Mar 22, 2022 at 4:16 PM Shrader, David Lee <dsh...@la...<mailto:dsh...@la...>> wrote:
Hey Robert,
8.6.16 does unload that "bad" modulefile, including unsetting the FOO env variable, and there is no longer the infinite loop when erroneously loading that "bad" modulefile twice.
I could see some use in trying to report an error upon unloading a modulefile, such as the modulefile no longer having enough information to do what it might need to. We have one like this where, if another required modulefile is unloaded before it (this particular setup does not use a modulefile hierarchy), it doesn't have enough information to construct the paths it needs to upon unloading, and it errors out because the paths aren't able to be removed from the required environment variables. An unload like the one done in 8.6.16 would unload the modulefile without actually removing the paths. Knowing that the removal failed in some way is nice to have. Of course, there is another solution to that scenario: store the needed paths from the load in an env variable and just use those if they are defined rather than try to reconstruct the path.
Thanks,
David
________________________________
From: Robert McLay <mc...@ta...<mailto:mc...@ta...>>
Sent: Tuesday, March 22, 2022 1:21 PM
To: Shrader, David Lee
Cc: lmod-users
Subject: Re: [EXTERNAL] Re: [Lmod-users] changes to tcl 'break' statement in lmod 8 series
Your example "bad" module shows why LmodBreak() in Lua (break in TCL) makes no sense when unloading. I have changed Lmod to ignore LmodBreak (or break in TCL) when unloading. Please set Lmod 8.6.16 to see if it works for you. The documentation has been updated as well.
Best,
Robert.
On Tue, Mar 22, 2022 at 11:40 AM Shrader, David Lee <dsh...@la...<mailto:dsh...@la...>> wrote:
Hey Robert,
Thank you for the 'module --config' command. I've used 'module --config' before; it is good to know that the TCL information is in there as well.
We went ahead and updated to 8.6.15, just to see how our tcl modulefiles behave, and they behave better than in 8.4.10. That is, we do not get the "unexpected symbol near '}' error when using a modulefile that pairs a 'puts stderr ...' command right before a break command. They also seem to properly stop loading at the point 'break' is called.
That being said, we did stumble on some strange behavior with a semi-bad modulefile (meaning, it shouldn't be used in any serious capacity) and a user-side mistaken use of 'module load'. In Lmod 7.8.16, we get errors for the aberrant behavior, but in Lmod 8.6.15, we get no errors and, in one case, an infinite loop.
It is probably easiest to just provide an example. Here is a "bad" modulefile ("bad" because it can never be unloaded):
$> cat foo3/1.0
#%Module
catch {set foo $env(FOO)}
if { [info exists foo] } {
puts stderr "already set"
break
}
setenv FOO "just me"
If this modulefile is loaded and then unloaded, Lmod 8.6.15 silently moves on, and the modulefile is not unloaded:
$> module load foo3/1.0
$> echo $FOO
just me
$> module rm foo3/1.0
already set
$> echo $FOO
just me
$> module list
Currently Loaded Modules:
1) foo3/1.0
In Lmod 7.8.16, we get this:
$> module load foo3/1.0
$> echo $FOO
just me
$> module rm foo3/1.0
already set
Lmod has detected the following error: /users/dshrader/modulefiles/common/foo3/1.0: (foo3/1.0):
While processing the following module(s):
Module fullname Module Filename
--------------- ---------------
foo3/1.0 /users/dshrader/modulefiles/common/foo3/1.0
$> module list
Currently Loaded Modules:
1) foo3/1.0
What's more, if one tries to load foo3/1.0 at this point (done completely by accident by myself, not through any intelligent choice 😉), Lmod 8.6.15 goes in to an infinite loop:
$> module list
Currently Loaded Modules:
1) foo3/1.0
$> module load foo3/1.0
already set
already set
already set
already set
already set
already set
...continues until crtl+c...
In Lmod 7.8.16, we get an error:
$> module list
Currently Loaded Modules:
1) foo3/1.0
$> module load foo3/1.0
already set
Lmod has detected the following error: /users/dshrader/modulefiles/common/foo3/1.0: (foo3/1.0):
While processing the following module(s):
Module fullname Module Filename
--------------- ---------------
foo3/1.0 /users/dshrader/modulefiles/common/foo3/1.0
So, it is possible to get Lmod 8.6.15 in to an interesting state. For myself, I do prefer the error messages from Lmod 7.8.16 over 8.6.15's current behavior. Since all of the above is based on a pretty poor modulefile, though, I'm not sure how much time should be spent on dealing with the above behavior. But, I wanted to at least let you know what we found as we were playing around with our tcl modulefiles under 8.6.15. For the most part, going to 8.6.15 looks like it will be fairly seamless for the existing TCL modulefiles, so we'll push for our OS vendors to update to 8.6.15 rather than stay at 8.4.10.
Thanks again!
David
________________________________
From: Robert McLay <mc...@ta...<mailto:mc...@ta...>>
Sent: Monday, March 21, 2022 4:49 PM
To: Shrader, David Lee
Cc: lmod-users
Subject: Re: [EXTERNAL] Re: [Lmod-users] changes to tcl 'break' statement in lmod 8 series
It is not a bug. There was no support for break in Lmod before 8.6+. It is just that having the tcl interpreter as a separate program meant that you got away with it.
> Do you mean that all of the 8.x versions include their own TCL interpreter, and 7.x relies on the system's TCL interpreter?
Yes
> We typically get Lmod from the OS vendor; is there a quick way to find out if the TCL interpreter was included?
$ module --config 2>&1 | grep -i "Use attached TCL"
Use attached TCL over system call yes
I would be best if you upgraded to Lmod 8.6.15 or later
Best,
Robert
On Mon, Mar 21, 2022 at 4:06 PM Shrader, David Lee <dsh...@la...<mailto:dsh...@la...>> wrote:
Hey Robert,
If I understand what you are saying, there should be no difference in how break is handled between 7.x and <8.6? If that is the case, then we may have hit a bug of some sort. But, since the latest version of Lmod is quite a ways beyond what we're looking to get (8.4.10), there may be little utility in trying to track it down. It might just be easier to try to get our upstream to provide an 8.6+...
Do you mean that all of the 8.x versions include their own TCL interpreter, and 7.x relies on the system's TCL interpreter?
We typically get Lmod from the OS vedor; is there a quick way to find out if the TCL interpreter was included?
Thanks again!
David
________________________________
From: Robert McLay <mc...@ta...<mailto:mc...@ta...>>
Sent: Monday, March 21, 2022 2:40 PM
To: Shrader, David Lee
Cc: lmod-users
Subject: [EXTERNAL] Re: [Lmod-users] changes to tcl 'break' statement in lmod 8 series
Well yes and no. There is no change in how break is or has been handled since forever. The change in supporting break is in Lmod 8.6+
The change is that by default Lmod now includes the TCL interpreter as part of the Lmod executable instead of executing a system call (/usr/bin/tclsh ....). You can configure Lmod to not include the tcl interpreter library as part of Lmod. That should give you the old behavior. Use --with-fastTCLInterp=no to configure Lmod without the tcl integration.
I would be interested in seeing how Lmod 8.6+ works with your modules even if you don't deploy it to users.
Best,
Robert.
On Mon, Mar 21, 2022 at 2:20 PM Shrader, David Lee via Lmod-users <lmo...@li...<mailto:lmo...@li...>> wrote:
Hello,
We're are potentially going from Lmod 7.8.16 to 8.4.10, and we have noticed that our tcl modulefiles that call "break" after a "puts stderr ..." message now generate something like the following error:
Lmod has detected the following error: Unable to load module because of error when evaluating modulefile: foo3
/modulefiles/foo3/1.0: [string "LmodMsgRaw([===[already set..."]:4: unexpected symbol near '}'
We have used this method of "puts stderr...; break" to exit out of a particular modulefile when an error occurs, such as a missing directory.
In looking over the readthedocs tcl function page, it looks like the "break" function has seen some attention, but we are looking at 8.4.10, not 8.6+. Did something change for the "break" command in 8.0 to 8.5 such that it won't work as we have been using it in 7.x?
Thank you very much for any and all help!
David
--
David Shrader
HPC-ENV High Performance Computing Systems
Los Alamos National Lab
_______________________________________________
Lmod-users mailing list
Lmo...@li...<mailto:Lmo...@li...>
https://lists.sourceforge.net/lists/listinfo/lmod-users<https://urldefense.com/v3/__https://nam12.safelinks.protection.outlook.com/?url=https*3A*2F*2Furldefense.com*2Fv3*2F__https*3A*2F*2Fnam12.safelinks.protection.outlook.com*2F*3Furl*3Dhttps*3A*2F*2Flists.sourceforge.net*2Flists*2Flistinfo*2Flmod-users*26data*3D04*7C01*7C*7C3ba8d73348fc47faf2a708da1da0a814*7C31d7e2a5bdd8414e9e97bea998ebdfe1*7C0*7C0*7C637854873104076991*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C3000*26sdata*3DJmgo4gnqYaZW4vTr5ocGcBhHKZWEevz4OdFq5xKP1Gw*3D*26reserved*3D0__*3BJSUlJSUlJSUlJSUlJSUlJSUl!!Bt8fGhp8LhKGRg!QOTIYbfOnk97D1Z-OL5Keb3b23zkF2CghPw_3vcxZpR2BHr5MxTmmUyumtmKsTac*24&data=04*7C01*7C*7Cd8836b92dcff4840e79f08da1e29df91*7C31d7e2a5bdd8414e9e97bea998ebdfe1*7C0*7C0*7C637855462444576759*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C3000&sdata=iNpfba4CJBaB2kF*2B6vVLzEs3H40st4FCH5bv2nBGIMc*3D&reserved=0__;JSUlJSUlJSUlJSUqKioqKiolJSoqKioqKioqKioqJSUqJSUlJSUlJSUlJSUlJSUlJSU!!Bt8fGhp8LhKGRg!QgKkt-CXAuMeybo6cv4QXRMn5XsvjqUVq_OPKc353yS62zQCvOylxbTxoi2CIkaC$>
>> This message is from an external sender. Learn more about why this <<
>> matters at https://links.utexas.edu/rtyclf<https://urldefense.com/v3/__https://nam12.safelinks.protection.outlook.com/?url=https*3A*2F*2Furldefense.com*2Fv3*2F__https*3A*2F*2Flinks.utexas.edu*2Frtyclf__*3B!!Bt8fGhp8LhKGRg!QOTIYbfOnk97D1Z-OL5Keb3b23zkF2CghPw_3vcxZpR2BHr5MxTmmUyumlv7Ge-H*24&data=04*7C01*7C*7Cd8836b92dcff4840e79f08da1e29df91*7C31d7e2a5bdd8414e9e97bea998ebdfe1*7C0*7C0*7C637855462444576759*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C3000&sdata=Iz9Pqca1VtOdzI8tWCBqs6ImP0gwNNg*2BbIYGNknJ60I*3D&reserved=0__;JSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUl!!Bt8fGhp8LhKGRg!QgKkt-CXAuMeybo6cv4QXRMn5XsvjqUVq_OPKc353yS62zQCvOylxbTxomI4UOvV$>. <<
--
Robert McLay, Ph.D.
Manager of Software Tools, HPC
mc...@ta...<mailto:mc...@ta...>
--
Robert McLay, Ph.D.
Manager of Software Tools, HPC
mc...@ta...<mailto:mc...@ta...>
--
Robert McLay, Ph.D.
Manager of Software Tools, HPC
mc...@ta...<mailto:mc...@ta...>
--
Robert McLay, Ph.D.
Manager of Software Tools, HPC
mc...@ta...<mailto:mc...@ta...>
--
Robert McLay, Ph.D.
Manager of Software Tools, HPC
mc...@ta...<mailto:mc...@ta...>
--
Robert McLay, Ph.D.
Manager of Software Tools, HPC
mc...@ta...<mailto:mc...@ta...>
|
