modules-interest — Discussion forum for modules usage.

[Modules] Reducing number of I/O operations

[Xavier D. <xav...@gm...>]

Hello All,

Modules 5.1 work is ongoing. A new configuration option, mcookie_check, is
introduced to skip the verification of each file in modulepath directories
to determine if they are modulefiles or not. When this option is set to
`eval`, file checks only occur when an attempt is made to evaluate these
files, not when listing available modules. So if you fully control what
files are saved under the modulepath directories you could get a nice speed
improvement, especially for commands like 'module avail':

https://modules.readthedocs.io/en/latest/MIGRATING.html#reducing-number-of-i-o-operations

A full cookbook recipe is also available to describe the features of
Modules that help to reduce the I/O load of walking through the available
modulefiles:

https://modules.readthedocs.io/en/latest/cookbook/reduce-io-load.html#reduce-io-load

Regards,
Xavier

Re: [Modules] environment modules for Conda environments

[Marius R. <mar...@es...>]

Dear Xavier,

Thank you for the detailed explanations and the updated behaviour of the 'puts' command. 

We will update to version 5.1 when it is out.

Best regards,
Marius

> On 9 Jan 2022, at 21:04, Xavier Delaruelle <xav...@gm...> wrote:
> 
> I have just added the ability to send content that will be evaluated
> in shell session prior any environment changes: using the puts command
> over the 'prestdout' channel [1].
> 
> This change will be included in version 5.1.
> 
> Best regards,
> Xavier
> 
> [1] https://modules.readthedocs.io/en/latest/modulefile.html#mfcmd-puts
> 
> Le mer. 5 janv. 2022 à 06:57, Xavier Delaruelle
> <xav...@gm...> a écrit :
>> 
>> Dear Marius, Dear Paul,
>> 
>> The content of puts statements are applied at the very end of the
>> produced environment change. These commands are run after all
>> environment change commands (generated by all evaluated modulefiles)
>> and whatever the position of the puts statement in the modulefile.
>> 
>> The workaround I found here is to mark the conda module sticky, with a
>> "module-tag sticky conda" command placed right above the "module load
>> conda" command. This way the conda module is not automatically
>> unloaded if it has been automatically loaded. This workaround requires
>> Modules 4.7 or above.
>> 
>> On the upcoming Modules 5.1, it will be possible to tag a module
>> "keep-loaded" to avoid the auto unload mechanism.
>> 
>> I have just created a new feature ticket to add the ability to put
>> such command at the start of the environment change commands rather at
>> the end [1].
>> 
>> I wish to everybody the best things for this new year.
>> 
>> Best regards,
>> Xavier
>> 
>> [1] https://github.com/cea-hpc/modules/issues/432
>> 
>> Le mar. 4 janv. 2022 à 16:53, Paul Markfort <pau...@gm...> a écrit :
>>> 
>>> Just a simple logic problem (you want to deactivate before you unload).
>>> 
>>> Try this:
>>> 
>>> if { [module-info mode remove] && ![module-info mode switch3] } {
>>>     puts stdout "conda deactivate"
>>> }
>>> 
>>> module load conda
>>> 
>>> if { [module-info mode load] || [module-info mode switch2] } {
>>>     puts stdout "conda activate $path"
>>> }
>>> 
>>> 
>>> 
>>> 
>>> 
>>> On 2022-01-04 04:43 AM, Marius Retegan wrote:
>>>> Dear all,
>>>> 
>>>> First of all happy new year and best wishes for 2022!
>>>> 
>>>> I would like to use environment modules to load/unload Conda environments (https://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html#managing-envs <https://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html#managing-envs>).
>>>> 
>>>> I have a working environment module for Conda alone. Now I need a module for a specific Conda environment. For this I use the code below to load/unload the environment:
>>>> 
>>>> ## My conda environment module file
>>>> 
>>>> module load conda
>>>> 
>>>> if { [module-info mode load] || [module-info mode switch2] } {
>>>>     puts stdout "conda activate $path"
>>>> } elseif { [module-info mode remove] && ![module-info mode switch3] } {
>>>>     puts stdout "conda deactivate"
>>>> }
>>>> 
>>>> If I do module load my_conda_environment; module unload my_conda_environment, I get an error because the conda dependency is unloaded first and then the conda command is not found anymore.
>>>> 
>>>> However this works: module load conda; module load my_conda_environment; module unload my_conda_environemt; module unload conda.
>>>> 
>>>> Is there a way to have the result of the previous line, without having to explicitly load conda? Thank you for your help.
>>>> 
>>>> Best regards,
>>>> Marius
> 
> 
> _______________________________________________
> Modules-interest mailing list
> Mod...@li...
> https://lists.sourceforge.net/lists/listinfo/modules-interest

Re: [Modules] environment modules for Conda environments

[Xavier D. <xav...@gm...>]

I have just added the ability to send content that will be evaluated
in shell session prior any environment changes: using the puts command
over the 'prestdout' channel [1].

This change will be included in version 5.1.

Best regards,
Xavier

[1] https://modules.readthedocs.io/en/latest/modulefile.html#mfcmd-puts

Le mer. 5 janv. 2022 à 06:57, Xavier Delaruelle
<xav...@gm...> a écrit :
>
> Dear Marius, Dear Paul,
>
> The content of puts statements are applied at the very end of the
> produced environment change. These commands are run after all
> environment change commands (generated by all evaluated modulefiles)
> and whatever the position of the puts statement in the modulefile.
>
> The workaround I found here is to mark the conda module sticky, with a
> "module-tag sticky conda" command placed right above the "module load
> conda" command. This way the conda module is not automatically
> unloaded if it has been automatically loaded. This workaround requires
> Modules 4.7 or above.
>
> On the upcoming Modules 5.1, it will be possible to tag a module
> "keep-loaded" to avoid the auto unload mechanism.
>
> I have just created a new feature ticket to add the ability to put
> such command at the start of the environment change commands rather at
> the end [1].
>
> I wish to everybody the best things for this new year.
>
> Best regards,
> Xavier
>
> [1] https://github.com/cea-hpc/modules/issues/432
>
> Le mar. 4 janv. 2022 à 16:53, Paul Markfort <pau...@gm...> a écrit :
> >
> > Just a simple logic problem (you want to deactivate before you unload).
> >
> > Try this:
> >
> > if { [module-info mode remove] && ![module-info mode switch3] } {
> >      puts stdout "conda deactivate"
> > }
> >
> > module load conda
> >
> > if { [module-info mode load] || [module-info mode switch2] } {
> >      puts stdout "conda activate $path"
> > }
> >
> >
> >
> >
> >
> > On 2022-01-04 04:43 AM, Marius Retegan wrote:
> > > Dear all,
> > >
> > > First of all happy new year and best wishes for 2022!
> > >
> > > I would like to use environment modules to load/unload Conda environments (https://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html#managing-envs <https://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html#managing-envs>).
> > >
> > > I have a working environment module for Conda alone. Now I need a module for a specific Conda environment. For this I use the code below to load/unload the environment:
> > >
> > > ## My conda environment module file
> > >
> > > module load conda
> > >
> > > if { [module-info mode load] || [module-info mode switch2] } {
> > >      puts stdout "conda activate $path"
> > > } elseif { [module-info mode remove] && ![module-info mode switch3] } {
> > >      puts stdout "conda deactivate"
> > > }
> > >
> > > If I do module load my_conda_environment; module unload my_conda_environment, I get an error because the conda dependency is unloaded first and then the conda command is not found anymore.
> > >
> > > However this works: module load conda; module load my_conda_environment; module unload my_conda_environemt; module unload conda.
> > >
> > > Is there a way to have the result of the previous line, without having to explicitly load conda? Thank you for your help.
> > >
> > > Best regards,
> > > Marius

Re: [Modules] environment modules for Conda environments

[Xavier D. <xav...@gm...>]

Dear Marius, Dear Paul,

The content of puts statements are applied at the very end of the
produced environment change. These commands are run after all
environment change commands (generated by all evaluated modulefiles)
and whatever the position of the puts statement in the modulefile.

The workaround I found here is to mark the conda module sticky, with a
"module-tag sticky conda" command placed right above the "module load
conda" command. This way the conda module is not automatically
unloaded if it has been automatically loaded. This workaround requires
Modules 4.7 or above.

On the upcoming Modules 5.1, it will be possible to tag a module
"keep-loaded" to avoid the auto unload mechanism.

I have just created a new feature ticket to add the ability to put
such command at the start of the environment change commands rather at
the end [1].

I wish to everybody the best things for this new year.

Best regards,
Xavier

[1] https://github.com/cea-hpc/modules/issues/432

Le mar. 4 janv. 2022 à 16:53, Paul Markfort <pau...@gm...> a écrit :
>
> Just a simple logic problem (you want to deactivate before you unload).
>
> Try this:
>
> if { [module-info mode remove] && ![module-info mode switch3] } {
>      puts stdout "conda deactivate"
> }
>
> module load conda
>
> if { [module-info mode load] || [module-info mode switch2] } {
>      puts stdout "conda activate $path"
> }
>
>
>
>
>
> On 2022-01-04 04:43 AM, Marius Retegan wrote:
> > Dear all,
> >
> > First of all happy new year and best wishes for 2022!
> >
> > I would like to use environment modules to load/unload Conda environments (https://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html#managing-envs <https://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html#managing-envs>).
> >
> > I have a working environment module for Conda alone. Now I need a module for a specific Conda environment. For this I use the code below to load/unload the environment:
> >
> > ## My conda environment module file
> >
> > module load conda
> >
> > if { [module-info mode load] || [module-info mode switch2] } {
> >      puts stdout "conda activate $path"
> > } elseif { [module-info mode remove] && ![module-info mode switch3] } {
> >      puts stdout "conda deactivate"
> > }
> >
> > If I do module load my_conda_environment; module unload my_conda_environment, I get an error because the conda dependency is unloaded first and then the conda command is not found anymore.
> >
> > However this works: module load conda; module load my_conda_environment; module unload my_conda_environemt; module unload conda.
> >
> > Is there a way to have the result of the previous line, without having to explicitly load conda? Thank you for your help.
> >
> > Best regards,
> > Marius

Re: [Modules] environment modules for Conda environments

[Paul M. <pau...@gm...>]

Just a simple logic problem (you want to deactivate before you unload).

Try this:

if { [module-info mode remove] && ![module-info mode switch3] } {
     puts stdout "conda deactivate"
}

module load conda

if { [module-info mode load] || [module-info mode switch2] } {
     puts stdout "conda activate $path"
}





On 2022-01-04 04:43 AM, Marius Retegan wrote:
> Dear all,
> 
> First of all happy new year and best wishes for 2022!
> 
> I would like to use environment modules to load/unload Conda environments (https://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html#managing-envs <https://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html#managing-envs>).
> 
> I have a working environment module for Conda alone. Now I need a module for a specific Conda environment. For this I use the code below to load/unload the environment:
> 
> ## My conda environment module file
> 
> module load conda
> 
> if { [module-info mode load] || [module-info mode switch2] } {
>      puts stdout "conda activate $path"
> } elseif { [module-info mode remove] && ![module-info mode switch3] } {
>      puts stdout "conda deactivate"
> }
> 
> If I do module load my_conda_environment; module unload my_conda_environment, I get an error because the conda dependency is unloaded first and then the conda command is not found anymore.
> 
> However this works: module load conda; module load my_conda_environment; module unload my_conda_environemt; module unload conda.
> 
> Is there a way to have the result of the previous line, without having to explicitly load conda? Thank you for your help.
> 
> Best regards,
> Marius
> 
> 
> 
> 
> _______________________________________________
> Modules-interest mailing list
> Mod...@li...
> https://lists.sourceforge.net/lists/listinfo/modules-interest
> 

-- 
--------------------------------------------------------
The views and opinions expressed above are strictly
those of the author(s).  The content of this message has
not been reviewed nor approved by any entity whatsoever.
--------------------------------------------------------
Paul FM         Info: http://paulfm.com/~paulfm/
--------------------------------------------------------

[Modules] environment modules for Conda environments

[Marius R. <mar...@es...>]

Dear all,

First of all happy new year and best wishes for 2022!

I would like to use environment modules to load/unload Conda environments (https://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html#managing-envs <https://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html#managing-envs>).

I have a working environment module for Conda alone. Now I need a module for a specific Conda environment. For this I use the code below to load/unload the environment:

## My conda environment module file

module load conda

if { [module-info mode load] || [module-info mode switch2] } {
    puts stdout "conda activate $path"
} elseif { [module-info mode remove] && ![module-info mode switch3] } {
    puts stdout "conda deactivate"
}

If I do module load my_conda_environment; module unload my_conda_environment, I get an error because the conda dependency is unloaded first and then the conda command is not found anymore. 

However this works: module load conda; module load my_conda_environment; module unload my_conda_environemt; module unload conda. 

Is there a way to have the result of the previous line, without having to explicitly load conda? Thank you for your help.

Best regards,
Marius

Re: [Modules] environment-modules, zsh and singularity problem

[Xavier D. <xav...@gm...>]

Hi,

I have just added new installaton options to define a specific
directory for the bash, fish and zsh completion script. When set for
zsh, the FPATH environment variable is not defined by Modules zsh
initialization
 script:

https://modules.readthedocs.io/en/latest/INSTALL.html#instopt---with-zshcompletiondir

This change will be part of v5.1. I will update the rpm spec file to
make use of these options right away.

Regards,
Xavier

Le mar. 19 oct. 2021 à 23:14, Xavier Delaruelle
<xav...@gm...> a écrit :
>
> Hi,
>
> I will look if I can improve the installation for the zsh function
> files not to rely on FPATH.
>
> However if the host environment is inherited within the container it
> implies that both environment should match to get something
> consistent. Especially if some module is loaded in the host prior
> running the container, all the environment variables set by loaded
> modules on host will get inherited in the container env. So FPATH is
> the visible part, but this issue affects every environment variable
> set on the host and some warning/error message could pop as soon as
> related software differ between host and container.
>
> Regards,
> Xavier
>
> Le mar. 19 oct. 2021 à 20:57, Götz Waschk <goe...@de...> a écrit :
> >
> > Am 19.10.21 um 19:15 schrieb Paul Markfort:
> > > I forgot to ask - what version of modules in installed with CentOS 8 (A
> > > search seems to suggest it is 4.5.2)?
> > > Keep in mind that the current version is 5.0.1
> >
> > Hi Paul,
> >
> > CentOS8 has 4.5.2, but I have tested 5.0.1 as well.
> >
> > > The page you reference has a warning about potential incompatibilities
> > > with modules.
> > > There is also an example on that page, on how to set an environment
> > > variable on the command line.
> > > Additionally - you could create a script (start-singularity.sh) that
> > > presets some environment variables before running singularity (since
> > > scripts run in a sub-shell - it won't affect your main shell's
> > > environment).
> >
> > That would work and I guess it is the right thing to do for certain
> > applications.
> >
> > > I think a possible fix (for you) is to just remove (remark out) the
> > > 'export FPATH' line in the  modules init/zsh script - it is worth a test
> > > (I note that FPATH is not exported by default in zsh - and that line is
> > > not in the init script for modules 4.1.4, but is there for 4.5.2 - I
> > > don't know which version made that change).
> > > I personally remark out the export lines in the init/bash script myself
> > > - as I don't see a need to export the module functions - as they
> > > generally get loaded again when a new interactive shell is started, and
> > > I don't want to accidentally depend on them in scripts.
> >
> > Thanks, I have just tested this, removing the "export FPATH" line did
> > the trick for me.
> >
> > Regards,
> > Götz
> >
> > --
> > Götz Waschk                            ° Phone:  +49 33762 77169
> > Deutsches Elektronen-Synchrotron DESY  ° E-Mail: goe...@de...
> > Platanenallee 6
> > 15738 Zeuthen Germany
> >
> > _______________________________________________
> > Modules-interest mailing list
> > Mod...@li...
> > https://lists.sourceforge.net/lists/listinfo/modules-interest

[Modules] Change modulefile command behavior

[Xavier D. <xav...@gm...>]

Hello All,

Modules 5.1 work is in progress. Just finished the introduction of new
options to the getenv, getvariant, remove-path, module unuse and
unsetenv modulefile commands to change their behavior on specific
evaluation mode. See:

https://modules.readthedocs.io/en/latest/MIGRATING.html#change-modulefile-command-behavior

This new feature set helps to address long-standing request to be able
to restore a modulepath that is unset at load time when unloading
modulefile:

https://github.com/cea-hpc/modules/issues/114

Regards,
Xavier

[Modules] Ongoing work : control unload behavior

[Xavier D. <xav...@gm...>]

Hello All,

I am currently working on adding the ability to choose the behavior of
the unsetenv, module unuse and remove-path commands when the
modulefile where these commands are used is evaluated in unload mode.

The design notes of these new features is available at:
https://modules.readthedocs.io/en/latest/design/control-mode-behaviors.html

Regards,
Xavier

[Modules] Modules development howtos

[Xavier D. <xav...@gm...>]

Hello All,

I have started to write guides for people looking at tweaking the
module command:

https://modules.readthedocs.io/en/latest/CONTRIBUTING.html#design-notes

Currently you will find a guide to introduce a new module sub-command
and another one to add a new configuration option.

Regards,
Xavier

Re: [Modules] environment-modules, zsh and singularity problem

[Xavier D. <xav...@gm...>]

Hi Everyone,

Please DO NOT CLICK on the 2 links provided in the previous email.

Adrien seems to have some issue with his mail box and both links must
be considered suspicious.

Regards,
Xavier

Le lun. 25 oct. 2021 à 12:56, Adrien COTTE - Groupe EOLEN
<adr...@eo...> a écrit :
>
> Hey there! You'll find all specification in the e-mail via the link lower:

Re: [Modules] environment-modules, zsh and singularity problem

[Paul M. <pau...@gm...>]

The serious issue is that the init script for zsh exports FPATH.
Eliminating that, may be a prudent (and simple) fix.

I suspect that the FPATH environment variable is not meant to be exported.
When I run zsh (without initializing variables), FPATH is set (and actually very long), but is definitely not exported (if I start bash from zsh, FPATH is no longer defined).





On 10/19/2021 4:14 PM, Xavier Delaruelle wrote:
> Hi,
> 
> I will look if I can improve the installation for the zsh function
> files not to rely on FPATH.
> 
> However if the host environment is inherited within the container it
> implies that both environment should match to get something
> consistent. Especially if some module is loaded in the host prior
> running the container, all the environment variables set by loaded
> modules on host will get inherited in the container env. So FPATH is
> the visible part, but this issue affects every environment variable
> set on the host and some warning/error message could pop as soon as
> related software differ between host and container.
> 
> Regards,
> Xavier
> 
> Le mar. 19 oct. 2021 à 20:57, Götz Waschk <goe...@de...> a écrit :
>>
>> Am 19.10.21 um 19:15 schrieb Paul Markfort:
>>> I forgot to ask - what version of modules in installed with CentOS 8 (A
>>> search seems to suggest it is 4.5.2)?
>>> Keep in mind that the current version is 5.0.1
>>
>> Hi Paul,
>>
>> CentOS8 has 4.5.2, but I have tested 5.0.1 as well.
>>
>>> The page you reference has a warning about potential incompatibilities
>>> with modules.
>>> There is also an example on that page, on how to set an environment
>>> variable on the command line.
>>> Additionally - you could create a script (start-singularity.sh) that
>>> presets some environment variables before running singularity (since
>>> scripts run in a sub-shell - it won't affect your main shell's
>>> environment).
>>
>> That would work and I guess it is the right thing to do for certain
>> applications.
>>
>>> I think a possible fix (for you) is to just remove (remark out) the
>>> 'export FPATH' line in the  modules init/zsh script - it is worth a test
>>> (I note that FPATH is not exported by default in zsh - and that line is
>>> not in the init script for modules 4.1.4, but is there for 4.5.2 - I
>>> don't know which version made that change).
>>> I personally remark out the export lines in the init/bash script myself
>>> - as I don't see a need to export the module functions - as they
>>> generally get loaded again when a new interactive shell is started, and
>>> I don't want to accidentally depend on them in scripts.
>>
>> Thanks, I have just tested this, removing the "export FPATH" line did
>> the trick for me.
>>
>> Regards,
>> Götz
>>
>> --
>> Götz Waschk                            ° Phone:  +49 33762 77169
>> Deutsches Elektronen-Synchrotron DESY  ° E-Mail: goe...@de...
>> Platanenallee 6
>> 15738 Zeuthen Germany
>>
>> _______________________________________________
>> Modules-interest mailing list
>> Mod...@li...
>> https://lists.sourceforge.net/lists/listinfo/modules-interest
> 
> 
> _______________________________________________
> Modules-interest mailing list
> Mod...@li...
> https://lists.sourceforge.net/lists/listinfo/modules-interest
> 

-- 
--------------------------------------------------------
The views and opinions expressed above are strictly
those of the author(s).  The content of this message has
not been reviewed nor approved by any entity whatsoever.
--------------------------------------------------------
Paul FM         Info: http://paulfm.com/~paulfm/
--------------------------------------------------------

Re: [Modules] environment-modules, zsh and singularity problem

[Xavier D. <xav...@gm...>]

Hi,

I will look if I can improve the installation for the zsh function
files not to rely on FPATH.

However if the host environment is inherited within the container it
implies that both environment should match to get something
consistent. Especially if some module is loaded in the host prior
running the container, all the environment variables set by loaded
modules on host will get inherited in the container env. So FPATH is
the visible part, but this issue affects every environment variable
set on the host and some warning/error message could pop as soon as
related software differ between host and container.

Regards,
Xavier

Le mar. 19 oct. 2021 à 20:57, Götz Waschk <goe...@de...> a écrit :
>
> Am 19.10.21 um 19:15 schrieb Paul Markfort:
> > I forgot to ask - what version of modules in installed with CentOS 8 (A
> > search seems to suggest it is 4.5.2)?
> > Keep in mind that the current version is 5.0.1
>
> Hi Paul,
>
> CentOS8 has 4.5.2, but I have tested 5.0.1 as well.
>
> > The page you reference has a warning about potential incompatibilities
> > with modules.
> > There is also an example on that page, on how to set an environment
> > variable on the command line.
> > Additionally - you could create a script (start-singularity.sh) that
> > presets some environment variables before running singularity (since
> > scripts run in a sub-shell - it won't affect your main shell's
> > environment).
>
> That would work and I guess it is the right thing to do for certain
> applications.
>
> > I think a possible fix (for you) is to just remove (remark out) the
> > 'export FPATH' line in the  modules init/zsh script - it is worth a test
> > (I note that FPATH is not exported by default in zsh - and that line is
> > not in the init script for modules 4.1.4, but is there for 4.5.2 - I
> > don't know which version made that change).
> > I personally remark out the export lines in the init/bash script myself
> > - as I don't see a need to export the module functions - as they
> > generally get loaded again when a new interactive shell is started, and
> > I don't want to accidentally depend on them in scripts.
>
> Thanks, I have just tested this, removing the "export FPATH" line did
> the trick for me.
>
> Regards,
> Götz
>
> --
> Götz Waschk                            ° Phone:  +49 33762 77169
> Deutsches Elektronen-Synchrotron DESY  ° E-Mail: goe...@de...
> Platanenallee 6
> 15738 Zeuthen Germany
>
> _______________________________________________
> Modules-interest mailing list
> Mod...@li...
> https://lists.sourceforge.net/lists/listinfo/modules-interest

Re: [Modules] Quarantine for a specific env var

[Adrien C. - G. E. <adr...@eo...>]

Many thanks, both of you.

Best,
Adrien

On Oct 19, 2021 21:36, Xavier Delaruelle <xav...@gm...> wrote:
Hello Adrien,

Quarantine mechanism is made to protect the execution of
modulecmd.tcl. It does not protect environment variable from being
defined/updated in your environment.

Maybe there is room for a PR here, with a new configuration option to
define environment variables that could not be set by modulefiles.

Best regards,
Xavier

Le lun. 18 oct. 2021 à 12:39, Adrien COTTE - Groupe EOLEN
<adr...@eo...> a écrit :
>
> Hello,
>
> For some reasons I have modules with "setenv PROMPT_COMMAND ..." calls breaking my prompt (liquidprompt).
>
> I tried to use "export MODULES_RUN_QUARANTINE=PROMPT_COMMAND" without success.
>
> How should I tell Modules to do not set a specific variable at load?
>
> Thanks in advance.
>
> Adrien,
> _______________________________________________
> Modules-interest mailing list
> Mod...@li...
> https://lists.sourceforge.net/lists/listinfo/modules-interest


_______________________________________________
Modules-interest mailing list
Mod...@li...
https://lists.sourceforge.net/lists/listinfo/modules-interest

Re: [Modules] Quarantine for a specific env var

[Xavier D. <xav...@gm...>]

Hello Adrien,

Quarantine mechanism is made to protect the execution of
modulecmd.tcl. It does not protect environment variable from being
defined/updated in your environment.

Maybe there is room for a PR here, with a new configuration option to
define environment variables that could not be set by modulefiles.

Best regards,
Xavier

Le lun. 18 oct. 2021 à 12:39, Adrien COTTE - Groupe EOLEN
<adr...@eo...> a écrit :
>
> Hello,
>
> For some reasons I have modules with "setenv PROMPT_COMMAND ..." calls breaking my prompt (liquidprompt).
>
> I tried to use "export MODULES_RUN_QUARANTINE=PROMPT_COMMAND" without success.
>
> How should I tell Modules to do not set a specific variable at load?
>
> Thanks in advance.
>
> Adrien,
> _______________________________________________
> Modules-interest mailing list
> Mod...@li...
> https://lists.sourceforge.net/lists/listinfo/modules-interest

Re: [Modules] environment-modules, zsh and singularity problem

[Götz W. <goe...@de...>]

Am 19.10.21 um 19:15 schrieb Paul Markfort:
> I forgot to ask - what version of modules in installed with CentOS 8 (A 
> search seems to suggest it is 4.5.2)?
> Keep in mind that the current version is 5.0.1

Hi Paul,

CentOS8 has 4.5.2, but I have tested 5.0.1 as well.

> The page you reference has a warning about potential incompatibilities 
> with modules.
> There is also an example on that page, on how to set an environment 
> variable on the command line.
> Additionally - you could create a script (start-singularity.sh) that 
> presets some environment variables before running singularity (since 
> scripts run in a sub-shell - it won't affect your main shell's 
> environment).

That would work and I guess it is the right thing to do for certain 
applications.

> I think a possible fix (for you) is to just remove (remark out) the 
> 'export FPATH' line in the  modules init/zsh script - it is worth a test 
> (I note that FPATH is not exported by default in zsh - and that line is 
> not in the init script for modules 4.1.4, but is there for 4.5.2 - I 
> don't know which version made that change).
> I personally remark out the export lines in the init/bash script myself 
> - as I don't see a need to export the module functions - as they 
> generally get loaded again when a new interactive shell is started, and 
> I don't want to accidentally depend on them in scripts.

Thanks, I have just tested this, removing the "export FPATH" line did 
the trick for me.

Regards,
Götz

-- 
Götz Waschk                            ° Phone:  +49 33762 77169
Deutsches Elektronen-Synchrotron DESY  ° E-Mail: goe...@de...
Platanenallee 6
15738 Zeuthen Germany

Re: [Modules] environment-modules, zsh and singularity problem

[Paul M. <pau...@gm...>]

I forgot to ask - what version of modules in installed with CentOS 8 (A search seems to suggest it is 4.5.2)?
Keep in mind that the current version is 5.0.1


The page you reference has a warning about potential incompatibilities with modules.
There is also an example on that page, on how to set an environment variable on the command line.
Additionally - you could create a script (start-singularity.sh) that presets some environment variables before running singularity (since scripts run in a sub-shell - it won't affect your main shell's environment).


I think a possible fix (for you) is to just remove (remark out) the 'export FPATH' line in the  modules init/zsh script - it is worth a test (I note that FPATH is not exported by default in zsh - and that line is not in the init script for modules 4.1.4, but is there for 4.5.2 - I don't know which version made that change).
I personally remark out the export lines in the init/bash script myself - as I don't see a need to export the module functions - as they generally get loaded again when a new interactive shell is started, and I don't want to accidentally depend on them in scripts.

Maybe Xavier could add an install option - that remarks out (most) export lines in all the initialization scripts when modules are installed.
Of course, remarking out the 'export FPATH' line in the zsh init script might be something to consider for all installations.






On 10/19/2021 10:36 AM, Götz Waschk wrote:
> Am 19.10.21 um 17:21 schrieb Paul Markfort:
>> This actually sounds like a bug with singularity - as your singularity session should be isolated from your normal environment.  This is like your shell variables affecting the environment in a virtual Linux machine.
>>
>> It seems to be a goal of singularity, that the process under it run under a completely isolated and consistent environment, see:
>> https://singularity.hpcng.org/
> 
> Dear Paul,
> 
> not really, the environment variables of the host are propagated to container environment unless you specify the option
>    -e, --cleanenv               clean environment before running container
> 
> https://singularity.hpcng.org/user-docs/3.8/environment_and_metadata.html#
> 
> It is just that $FPATH is a special variable for zsh. It would be nice if you could mask specific vars in singularity though.
> 
> Regards,
> Götz
> 
> 
> _______________________________________________
> Modules-interest mailing list
> Mod...@li...
> https://lists.sourceforge.net/lists/listinfo/modules-interest
> 

-- 
--------------------------------------------------------
The views and opinions expressed above are strictly
those of the author(s).  The content of this message has
not been reviewed nor approved by any entity whatsoever.
--------------------------------------------------------
Paul FM         Info: http://paulfm.com/~paulfm/
--------------------------------------------------------

Re: [Modules] environment-modules, zsh and singularity problem

[Götz W. <goe...@de...>]

Am 19.10.21 um 17:21 schrieb Paul Markfort:
> This actually sounds like a bug with singularity - as your singularity 
> session should be isolated from your normal environment.  This is like 
> your shell variables affecting the environment in a virtual Linux machine.
> 
> It seems to be a goal of singularity, that the process under it run 
> under a completely isolated and consistent environment, see:
> https://singularity.hpcng.org/

Dear Paul,

not really, the environment variables of the host are propagated to 
container environment unless you specify the option
   -e, --cleanenv               clean environment before running container

https://singularity.hpcng.org/user-docs/3.8/environment_and_metadata.html#

It is just that $FPATH is a special variable for zsh. It would be nice 
if you could mask specific vars in singularity though.

Regards,
Götz
-- 
Götz Waschk                            ° Phone:  +49 33762 77169
Deutsches Elektronen-Synchrotron DESY  ° E-Mail: goe...@de...
Platanenallee 6
15738 Zeuthen Germany

Re: [Modules] environment-modules, zsh and singularity problem

[Paul M. <pau...@gm...>]

This actually sounds like a bug with singularity - as your singularity session should be isolated from your normal environment.  This is like your shell variables affecting the environment in a virtual Linux machine.

It seems to be a goal of singularity, that the process under it run under a completely isolated and consistent environment, see:
https://singularity.hpcng.org/


On 10/19/2021 9:35 AM, Götz Waschk wrote:
> Hi everyone,
> 
> I have noticed a problem with environment-modules since upgrading to CentOS8. The startup file /usr/share/Modules/init/zsh modifies the variable $FPATH that defines the function search path used by zsh. If the variable is modified from its default, it is inherited by singularity when running a container, breaking functionality if this container has a different version of zsh.
> 
> Here's an example of this behaviour:
> 
> [wgs03] ~ % echo $FPATH
> /usr/share/Modules/init/zsh-functions:/usr/local/share/zsh/site-functions:/usr/share/zsh/site-functions:/usr/share/zsh/5.5.1/functions
> [wgs03] ~ %  singularity run /project/singularity/images/SL6.img
> ************************************************************************
> * You are now running a shell in a  Singularity container providing a  *
> * Scientific Linux 6 environment.  Most things should just work as if  *
> * you had just logged in to such a system, but there are differences.  *
> ************************************************************************
> /etc/site/zshrc:72: compinit: function definition file not found
> [wgs03:SL6] ~ % echo $FPATH
> /usr/share/Modules/init/zsh-functions:/usr/local/share/zsh/site-functions:/usr/share/zsh/site-functions:/usr/share/zsh/5.5.1/functions
> 
> If I remove environment-modules from the system, this problem doesn't occur:
> 
> [wgs03] ~ % echo $FPATH
> /usr/local/share/zsh/site-functions:/usr/share/zsh/site-functions:/usr/share/zsh/5.5.1/functions
> [wgs03] ~ %  singularity run /project/singularity/images/SL6.img
> ************************************************************************
> * You are now running a shell in a  Singularity container providing a  *
> * Scientific Linux 6 environment.  Most things should just work as if  *
> * you had just logged in to such a system, but there are differences.  *
> ************************************************************************
> [wgs03:SL6] ~ % echo $FPATH
> /usr/share/zsh/site-functions:/usr/share/zsh/4.3.11/functions
> 
> So every time, $FPATH is modified, it is inherited by the container. I'd prefer it if zsh wouldn't modify $FPATH. Wouldn't it work as well if _module was placed in /usr/share/zsh/site-functions? My test regarding this was fine.
> 
> Regards,
> Götz Waschk
> 
> 
> 
> _______________________________________________
> Modules-interest mailing list
> Mod...@li...
> https://lists.sourceforge.net/lists/listinfo/modules-interest
> 

-- 
--------------------------------------------------------
The views and opinions expressed above are strictly
those of the author(s).  The content of this message has
not been reviewed nor approved by any entity whatsoever.
--------------------------------------------------------
Paul FM         Info: http://paulfm.com/~paulfm/
--------------------------------------------------------

[Modules] environment-modules, zsh and singularity problem

[Götz W. <goe...@de...>]

Hi everyone,

I have noticed a problem with environment-modules since upgrading to 
CentOS8. The startup file /usr/share/Modules/init/zsh modifies the 
variable $FPATH that defines the function search path used by zsh. If 
the variable is modified from its default, it is inherited by 
singularity when running a container, breaking functionality if this 
container has a different version of zsh.

Here's an example of this behaviour:

[wgs03] ~ % echo $FPATH
/usr/share/Modules/init/zsh-functions:/usr/local/share/zsh/site-functions:/usr/share/zsh/site-functions:/usr/share/zsh/5.5.1/functions
[wgs03] ~ %  singularity run /project/singularity/images/SL6.img
************************************************************************
* You are now running a shell in a  Singularity container providing a  *
* Scientific Linux 6 environment.  Most things should just work as if  *
* you had just logged in to such a system, but there are differences.  *
************************************************************************
/etc/site/zshrc:72: compinit: function definition file not found
[wgs03:SL6] ~ % echo $FPATH
/usr/share/Modules/init/zsh-functions:/usr/local/share/zsh/site-functions:/usr/share/zsh/site-functions:/usr/share/zsh/5.5.1/functions

If I remove environment-modules from the system, this problem doesn't occur:

[wgs03] ~ % echo $FPATH
/usr/local/share/zsh/site-functions:/usr/share/zsh/site-functions:/usr/share/zsh/5.5.1/functions
[wgs03] ~ %  singularity run /project/singularity/images/SL6.img
************************************************************************
* You are now running a shell in a  Singularity container providing a  *
* Scientific Linux 6 environment.  Most things should just work as if  *
* you had just logged in to such a system, but there are differences.  *
************************************************************************
[wgs03:SL6] ~ % echo $FPATH
/usr/share/zsh/site-functions:/usr/share/zsh/4.3.11/functions

So every time, $FPATH is modified, it is inherited by the container. I'd 
prefer it if zsh wouldn't modify $FPATH. Wouldn't it work as well if 
_module was placed in /usr/share/zsh/site-functions? My test regarding 
this was fine.

Regards,
Götz Waschk

-- 
Götz Waschk                            ° Phone:  +49 33762 77169
Deutsches Elektronen-Synchrotron DESY  ° E-Mail: goe...@de...
Platanenallee 6
15738 Zeuthen Germany

Re: [Modules] Quarantine for a specific env var

[Sternberg, M. G. <ste...@an...>]

Adrien,

Liquidprompt <https://liquidstate.net/liquid-prompt/> is touchy regarding PROMPT_COMMAND, and unsurprisingly so. A warning at https://liquidprompt.readthedocs.io/en/stable/install.html#shell-installation says "Do not export PROMPT_COMMAND", and that puts it outside the purview of Environment Modules. Prompt capabilities and configurations are highly shell-specific, making them rather incongruous with modules.

It'd be best to configure Liquidprompt directly in the shell you use, or source a common Bourne shell script if you run both shells supported (bash+zsh).


Michael


> On Oct 18, 2021, at 05:39, Adrien COTTE - Groupe EOLEN <adr...@eo...> wrote:
> For some reasons I have modules with "setenv PROMPT_COMMAND ..." calls breaking my prompt (liquidprompt).
> I tried to use "export MODULES_RUN_QUARANTINE=PROMPT_COMMAND" without success.
> How should I tell Modules to do not set a specific variable at load?

[Modules] Quarantine for a specific env var

[Adrien C. - G. E. <adr...@eo...>]

Hello,

For some reasons I have modules with "setenv PROMPT_COMMAND ..." calls breaking my prompt (liquidprompt).

I tried to use "export MODULES_RUN_QUARANTINE=PROMPT_COMMAND" without success.

How should I tell Modules to do not set a specific variable at load?

Thanks in advance.

Adrien,

[Modules] Modules 5.0.1 released

[Xavier D. <xav...@gm...>]

Hello All,

Modules 5.0.1 is out. This release brings fixes for the few bugs found
on 5.0.0 and some documentation improvement.

The changes introduced in this bugfix release are detailed at:

 https://modules.readthedocs.io/en/stable/NEWS.html#modules-5-0-1-2021-10-16

Details on the new features introduced in the 5.0 series are provided
in the MIGRATING document.

 http://modules.readthedocs.io/en/stable/MIGRATING.html

The tarball of this new version can be downloaded at:

 http://downloads.sourceforge.net/modules/modules-5.0.1.tar.gz

The zipball to install this new version on Windows platform can be
downloaded at:

 http://downloads.sourceforge.net/modules/modules-5.0.1-win.zip

If you encounter any issue, please let us know by creating a ticket on the
project bug tracker at:

 https://github.com/cea-hpc/modules/issues

Special thanks to all the people who reported issues.

Best regards,
Xavier

Re: [Modules] Parsing module result

[Xavier D. <xav...@gm...>]

Hi Michael,

I have just added an issue to update sh-to-mod sub-command and
source-sh on version 5.1 and filter-out the environment variables sets
by Modules which are intended for its own internal use.

https://github.com/cea-hpc/modules/issues/427

Best regards,
Xavier

Le jeu. 14 oct. 2021 à 14:33, Sternberg, Michael G.
<ste...@an...> a écrit :
>
> Xavier,
>
> Indeed, thank you for integrating this!
>
> Now that "module sh-to-mod" is implemented internal to modulecmd.tcl, an interesting question and opportunity comes up:  Would it be useful – or even necessary for consistency – to have sh-to-mod *filter out* (i.e., not issue) modifications of module-internal variables, like I did pragmatically with grep?  Such internals would not be of interest for later deployment of the generated module, and make for an "Inception" challenge to modulecmd on eventual load. Having module invocations already occur in the input is certainly an edge case, but one that could well come up in porting situations.
>
> Michael Strelnikov: Could you illustrate the motivation of your question?
>
>
> With best regards,
> Michael
>
>
> > On Oct 13, 2021, at 23:50, Xavier Delaruelle <xav...@gm...> wrote:
> > […] Please note that the createmodule.py script has been replaced by the
> > "sh-to-mod" sub-command of module:
> > https://modules.readthedocs.io/en/latest/MIGRATING.html#sh-to-mod-sub-command
> > Regards,
> > Xavier
> >
> > Le jeu. 14 oct. 2021 à 02:10, Sternberg, Michael G. via
> > Modules-interest <mod...@li...> a écrit :
> >>
> >> […] you could let Modules itself do the work of gathering and interpreting the accumulated changes of environment vars:
> >>    echo module load foo > /tmp/foo.sh
> >>    createmodule.py /tmp/foo.sh \
> >>        | grep -Ev '(_modshare|MODULES_LM|_LMFILES_|LOADEDMODULES)'
>

Re: [Modules] Reorganization suggestion for modules/{version}/init directory

[Xavier D. <xav...@gm...>]

I will add for v5.1 a new state named 'supported_shells' and a state
sub-command to report current state value:

$ module state supported_shells
Modules Release 5.0.0+17-gbbab6f4a (2021-10-13)

- State name -----------.- Value ----------------------------------------------
supported_shells         sh bash ksh zsh csh tcsh fish cmd tcl perl
python ruby lisp cmake r

See https://github.com/cea-hpc/modules/issues/426

Xavier

Le ven. 15 oct. 2021 à 17:43, Xav...@CE...
<Xav...@ce...> a écrit :
>
> RedHat has moved to Modules 4 for EL8, and if everything goes as
> expected Modules 5 should be part of EL9.
>
> openSUSE Leap is still relying on Modules 3.2, but openSUSE
> Tumbleweed has version 4.8.
>
> The repology website is really helpful to learn how Modules is
> available across OSes:
>
> https://repology.org/project/environment-modules/versions
>
> Globally the new Modules versions are available on recent OS
> releases.
>
> Xavier
>
>
> On Fri, 2021-10-15 at 10:28 -0500, Paul Markfort wrote:
> > That works (the profile.d stuff should still be put together).
> >
> > Maybe add a psuedo shell,  "list-all-shells"
> > Output is the list of shells in a machine parse-able format (so the
> > output could be used in shells).   Or that could be the output
> > returned if there are no arguments.
> >
> >
> > Now we have to figure out why Redhat and SUSE both seem to be stuck
> > on modules 3.2.10
> >
> > If it is a (perceived) performance issue, perhaps you need to bring
> > back a binary version - in the form of the modulecmd.tcl command is
> > compiled into modulecmd.bin (in the libexec folder) - or include
> > instructions on how to do that.
> > There are good reasons why most people would want to use the tcl
> > script (cross platform compatibility) while others may want a
> > binary (performance).
> >
> >
> >
> > On 10/15/2021 12:11 AM, Xavier Delaruelle wrote:
> > > Hi Paul,
> > >
> > > Current organization of the init directory already gives a good
> > > insight of the shells that are supported. Except for the profile
> > > scripts, the first word of each file designates the shell.
> > > Splitting
> > > things with more directories will make it harder to get a full
> > > overview. Also such change is a major one and v5.0 is still new
> > > so
> > > such file movement cannot happen before 6.0.
> > >
> > > What could be done instead is to list the supported shells when
> > > no
> > > shell or an unsupported shell is set on $MODULES_CMD :
> > >
> > > $ $MODULES_CMD
> > > ERROR: Unknown shell type ''
> > >    Supported shells are: sh, bash, ksh, zsh, csh, tcsh, fish,
> > > cmd, tcl, perl,
> > >     python, ruby, lisp, cmake and r
> > >
> > > Regards,
> > > Xavier
> > >
> > > Le jeu. 14 oct. 2021 à 17:50, Paul Markfort <pau...@gm...>
> > > a écrit :
> > > >
> > > >
> > > > Xavier,
> > > >
> > > > May I suggest a slight reorganization of the
> > > > "modules/{version}/init" folder.
> > > >
> > > > I note that the contents of this folder, is the easiest way to
> > > > get a list of shells that a particular version of modules
> > > > supports.
> > > >
> > > >
> > > > First (and probably important), move profile.csh and profile.sh
> > > > to a subfolder  profile.d (and rename them modules.*).
> > > > This will make them easier to find and just use these by
> > > > copying them to /etc/profile.d/
> > > > (I also suggest you put a test in those scripts that they are
> > > > effectively noop if the user is root).
> > > >
> > > > Next, rename the *_* and *-* files and folders to *.* - adding
> > > > a .d to the folders (tcsh.completion, zsh.functions.d ) - or
> > > > create a subfolder (maybe auxiliary.d, misc.d, or extra.d) for
> > > > all of them.  This part will make it easier to use the list of
> > > > files to determine what shells modules supports (both by sight,
> > > > and particularly in scripts).
> > > >
> > > >
> > > > Thanks.
> > > >
> > > >
> > > > --
> > > > --------------------------------------------------------
> > > > The views and opinions expressed above are strictly
> > > > those of the author(s).  The content of this message has
> > > > not been reviewed nor approved by any entity whatsoever.
> > > > --------------------------------------------------------
> > > > Paul FM         Info: http://paulfm.com/~paulfm/
> > > > --------------------------------------------------------
> > > >
> > > >
> > > > _______________________________________________
> > > > Modules-interest mailing list
> > > > Mod...@li...
> > > > https://lists.sourceforge.net/lists/listinfo/modules-interest
> > >
> > >
> > > _______________________________________________
> > > Modules-interest mailing list
> > > Mod...@li...
> > > https://lists.sourceforge.net/lists/listinfo/modules-interest
> > >
> >
> >
>
> _______________________________________________
> Modules-interest mailing list
> Mod...@li...
> https://lists.sourceforge.net/lists/listinfo/modules-interest