modules-interest — Discussion forum for modules usage.

[Modules] Environment Modules for Multi-arch Environment

[Michael K. <mi...@do...>]

Folks,

I have the following setup:

/net/software/{x86_64,aarch64,armv7l,riscv} contains all software on 
shared storage.  /net/software/modules/modulefiles contains the 
corresponding module files for these software packages, e.g., 
clang/13.0.0, gcc/9.2.0, etc.  The module files use the $ARCH 
environment variable to setup the environment for the software.

Since I do not have all software packages compiled for all systems, I 
was wondering what would be the best way to avoid loading (or even 
better hide) modules that do not have the software package for a 
particular architecture?

I have come up with either checking in the module file, if the software 
exists and if not throw some sort of error.  I could also have 
$ARCH-specific module locations and use "module use" on these locations, 
depending on the machine I'm on.  But is there a better way to automate 
all this w/o too much duplication of module files or code in the module 
files?

I'm using Modules Release 4.4.1 (2020-01-03), all systems are Ubuntu 
systems.

Kind regards,
         -michael

[Modules] Does a linter for module files exist?

[Erik P. <eri...@si...>]

Hi,

I’m managing a lot of module files and I was wondering if there is a linter for modules files or maybe some formatter (espeically for indentation). I have seen that nagelfar132 is a good linter for TCL code and that you can create a new syntax database using a provided scrip, but I can’t figure out if I can even run that script with the module interpreter.

Suggestions are welcome!

Regards,
Erik Parawell

Re: [Modules] Reducing number of I/O operations

[Xavier D. <xav...@gm...>]

Hello All,

Following this discussion, I would like to let you know that I have added a
"Configuration options" summary table in the INSTALL document:

https://modules.readthedocs.io/en/latest/INSTALL.html#configuration-options

The description of configuration options, installation options and
environment variables have also been enhanced to report the relations
between each items.

Regards,
Xavier

Le sam. 15 janv. 2022 à 05:21, Xavier Delaruelle <
xav...@gm...> a écrit :

> Hi Paul,
>
> The configuration option should be looked at rather its corresponding
> superseding environment variable. Module configuration option has default
> value and can then be superseded by setting a MODULE_* environment
> variable. Configuration option handling can be controlled with "module
> config" command. By default it lists all known configuration options:
>
> $ module config
> Modules Release 5.0.1+221-g754913fe (2022-01-10)
>
> - Config. name ---------.- Value (set by if default overridden)
> ---------------
> advanced_version_spec     1
> auto_handling             1 (env-var)
> avail_indepth             1
> ...
>
> When using the default value set when Modules is built (through the
> ./configure script), module config will just report this value:
>
> $ module config mcookie_check
> Modules Release 5.0.1+221-g754913fe (2022-01-10)
>
> - Config. name ---------.- Value (set by if default overridden)
> ---------------
> mcookie_check             always
>
> You can also use this command to set a specific value to the config option
> (it will set the corresponding MODULES_* environment variable):
>
> $ module config mcookie_check eval
>
> When the default value is superseded, a '(env-var)' mention appear:
>
> $ module config mcookie_check
> Modules Release 5.0.1+221-g754913fe (2022-01-10)
>
> - Config. name ---------.- Value (set by if default overridden)
> ---------------
> mcookie_check             eval (env-var)
>
> The initrc file, evaluated when the module command initializes in user's
> shell session, can host "module config" commands. This is the good location
> to define your site-specific configuration. See the configuration section
> of the installation guide:
>
> https://modules.readthedocs.io/en/latest/INSTALL.html#configuration
>
> I will look at better explaining the default value of configuration option
> in the documentation.
>
> Best regards,
> Xavier
>
>
>
> Le ven. 14 janv. 2022 à 17:37, Paul Markfort <pau...@gm...> a
> écrit :
>
>> Note:
>>         module config --dump-state
>>
>> only lists the environment variables that are set. Unless there is
>> another command that does this, maybe add a "module config -dump-state-all"
>> that lists all the environment variables that could be set and the current
>> assumed values (or better, or additionally, a -dump-state-unset to only
>> list those that are not set).
>>
>>
>> On 2022-01-14 10:23 AM, Paul Markfort wrote:
>> > Great to see this improvement.
>> >
>> > Do you have a reference on how to change the assumed default values for
>> any of the MODULES_* environment variables (the value assumed if the
>> environment variable is not set) in the siteconfig.tcl file?  In
>> particular, I would want to have MODULES_MCOOKIE_CHECK assumed to be eval
>> (for the site, not in the distribution); but, there are others that might
>> be worthwhile to have a site default for as well.
>> >
>> > Is there a module command that can list all these environment variables
>> and what the current (assumed or set) setting is for each?  Perhaps that
>> command (if it exists) should be listed on the documentation pages just
>> below the ENVIRONMENT heading somehow.
>> >
>> >
>> > Also, in the documentation of the environment variables, the default
>> assumed values for many of them is not listed.  May I recommend that the
>> default value be listed right after the variable either like this:
>> > MODULES_MCOOKIE_CHECK        default_value: always
>> >
>> > or:
>> > MODULES_MCOOKIE_CHECK
>> >          default_value: always
>> >
>> >
>> > On 2022-01-14 12:41 AM, Xavier Delaruelle wrote:
>> >> Hello All,
>> >>
>> >> Modules 5.1 work is ongoing. A new configuration option,
>> mcookie_check, is
>> >> introduced to skip the verification of each file in modulepath
>> directories
>> >> to determine if they are modulefiles or not. When this option is set to
>> >> `eval`, file checks only occur when an attempt is made to evaluate
>> these
>> >> files, not when listing available modules. So if you fully control what
>> >> files are saved under the modulepath directories you could get a nice
>> speed
>> >> improvement, especially for commands like 'module avail':
>> >>
>> >>
>> https://modules.readthedocs.io/en/latest/MIGRATING.html#reducing-number-of-i-o-operations
>> >>
>> >> A full cookbook recipe is also available to describe the features of
>> >> Modules that help to reduce the I/O load of walking through the
>> available
>> >> modulefiles:
>> >>
>> >>
>> https://modules.readthedocs.io/en/latest/cookbook/reduce-io-load.html#reduce-io-load
>> >>
>> >> Regards,
>> >> Xavier
>> >>
>> >>
>> >>
>> >> _______________________________________________
>> >> Modules-interest mailing list
>> >> Mod...@li...
>> >> https://lists.sourceforge.net/lists/listinfo/modules-interest
>> >>
>> >
>>
>> --
>> --------------------------------------------------------
>> The views and opinions expressed above are strictly
>> those of the author(s).  The content of this message has
>> not been reviewed nor approved by any entity whatsoever.
>> --------------------------------------------------------
>> Paul FM         Info: http://paulfm.com/~paulfm/
>> --------------------------------------------------------
>>
>>
>> _______________________________________________
>> Modules-interest mailing list
>> Mod...@li...
>> https://lists.sourceforge.net/lists/listinfo/modules-interest
>>
>

[Modules] Improvement on sh-to-mod/source-sh mechanisms

[Xavier D. <xav...@gm...>]

Hello All,

sh-to-mod sub-command and source-sh modulefile command are mechanisms to
translate the environment changes of a shell script into a modulefile.
After the recent support of shell completion definition, several fixes have
been made on these mechanisms:

* Support nested function definition on fish shell.
* Correctly detect functions with a _ character in their name on fish
shell.
* Update the module shell function definition for sh-kind shell to enclose
the output generated by modulecmd.tcl in quotes in order to pass it to the
eval command. This change fixes definition of shell function
* Fix csh shell alias detection when enclosed in parenthesis.
* Include for fish shell environment analysis the private functions defined
by evaluated script, in order to catch all definitions that may be useful
for shell completion.
* Correctly detect empty function on fish shell.

With all these improvements, shell function are better handled when
evaluating complex shell scripts.

Tests have been added to track any regression of these mechanisms against
well known setup env scripts (Spack, Miniconda, OpenFOAM)

All of these improvements will be available in upcoming version 5.1.

Regards,
Xavier

Re: [Modules] Building perl with CPAN modules for modules

[Harlan S. <ha...@pf...>]

Thanks, Xavier!

On 1/28/2022 11:10 PM, Xavier Delaruelle wrote:
> Hi Harlan,
> 
> There are two popular build tools in the scientific software field that 
> build software and a modulefile to use it: Spack [1] and EasyBuild [2].
> 
> I have quickly look at both and I do not find track of openssl in the 
> build recipe for Perl on Spack, but in EasyBuild the openssl dependency 
> is clearly expressed.
> 
> So I would suggest you to do a small test setup of EasyBuild, try to 
> build Perl and some of its CPAN modules (like Net::SSLeay that clearly 
> needs openssl) and look at the obtained modulefiles and at how things 
> are built.
> 
> Just looking at the build recipe for Net::SSLeay [3], there is some 
> pre-config option define:
> 
>          'preconfigopts': "export OPENSSL_PREFIX=$EBROOTOPENSSL && ",
> 
> Regards,
> Xavier
> 
> [1] https://spack.io/ <https://spack.io/>
> [2] https://easybuild.io/ <https://easybuild.io/>
> [3] 
> https://github.com/easybuilders/easybuild-easyconfigs/blob/develop/easybuild/easyconfigs/p/Perl/Perl-5.34.0-GCCcore-11.2.0.eb 
> <https://github.com/easybuilders/easybuild-easyconfigs/blob/develop/easybuild/easyconfigs/p/Perl/Perl-5.34.0-GCCcore-11.2.0.eb>
> 
> Le ven. 28 janv. 2022 à 12:57, Harlan Stenn <ha...@pf... 
> <mailto:ha...@pf...>> a écrit :
> 
>     Is there a place where folks are collecting how to build various tools
>     to run under modules?
> 
>     For example, perl and its CPAN modules need to have a path to the
>     version of openssl that they are built against.
> 
>     The things I did years ago to build perl-X against openssl-Y and have
>     the result be usable by modules no longer work...
> 
>     I'm looking for the current build incantation for perl that will get
>     the
>     desired openssl headers and libraries at the front of the appropriate
>     search paths.
> 
>     H
> 
> 
>     _______________________________________________
>     Modules-interest mailing list
>     Mod...@li...
>     <mailto:Mod...@li...>
>     https://lists.sourceforge.net/lists/listinfo/modules-interest
>     <https://lists.sourceforge.net/lists/listinfo/modules-interest>
> 
> 
> 
> _______________________________________________
> Modules-interest mailing list
> Mod...@li...
> https://lists.sourceforge.net/lists/listinfo/modules-interest

Re: [Modules] Building perl with CPAN modules for modules

[Xavier D. <xav...@gm...>]

Hi Harlan,

There are two popular build tools in the scientific software field that
build software and a modulefile to use it: Spack [1] and EasyBuild [2].

I have quickly look at both and I do not find track of openssl in the build
recipe for Perl on Spack, but in EasyBuild the openssl dependency is
clearly expressed.

So I would suggest you to do a small test setup of EasyBuild, try to build
Perl and some of its CPAN modules (like Net::SSLeay that clearly needs
openssl) and look at the obtained modulefiles and at how things are built.

Just looking at the build recipe for Net::SSLeay [3], there is some
pre-config option define:

        'preconfigopts': "export OPENSSL_PREFIX=$EBROOTOPENSSL && ",

Regards,
Xavier

[1] https://spack.io/
[2] https://easybuild.io/
[3]
https://github.com/easybuilders/easybuild-easyconfigs/blob/develop/easybuild/easyconfigs/p/Perl/Perl-5.34.0-GCCcore-11.2.0.eb

Le ven. 28 janv. 2022 à 12:57, Harlan Stenn <ha...@pf...> a écrit :

> Is there a place where folks are collecting how to build various tools
> to run under modules?
>
> For example, perl and its CPAN modules need to have a path to the
> version of openssl that they are built against.
>
> The things I did years ago to build perl-X against openssl-Y and have
> the result be usable by modules no longer work...
>
> I'm looking for the current build incantation for perl that will get the
> desired openssl headers and libraries at the front of the appropriate
> search paths.
>
> H
>
>
> _______________________________________________
> Modules-interest mailing list
> Mod...@li...
> https://lists.sourceforge.net/lists/listinfo/modules-interest
>

[Modules] Building perl with CPAN modules for modules

[Harlan S. <ha...@pf...>]

Is there a place where folks are collecting how to build various tools 
to run under modules?

For example, perl and its CPAN modules need to have a path to the 
version of openssl that they are built against.

The things I did years ago to build perl-X against openssl-Y and have 
the result be usable by modules no longer work...

I'm looking for the current build incantation for perl that will get the 
desired openssl headers and libraries at the front of the appropriate 
search paths.

H

Re: [Modules] Reducing number of I/O operations

[Sternberg, M. G. <ste...@an...>]

Xavier,

To close the loop on "mcookie": I suppose the terminology is not enough of a worry to make a change.

For background: Certainly the term "magic cookie" dates back to 3.x and earlier. I didn't follow the internals of 3.x and 4.x well enough, and spoke up because I had incidentally just looked up file(1) and magic(5), and suddenly realized that the "magic" had been lost[*] over the years.  :-)   Looking over a few historic RPM versions that I had on hand, I see that "mcookie" – as opposed to "magic cookie" – was code-internal first, introduced in the 4.2 testsuite, then was surfaced as a user-visible feature in the form of the configuration item "mcookie_version_check"  in version 4.7.


With best regards,
Michael


[*] Not of environment-modules as a project, I hasten to add. I much appreciate the recent and feature-rich developments driven by Xavier.


> On Jan 16, 2022, at 01:23, Xavier Delaruelle <xav...@gm...> wrote:
> This terminology "Modules magic cookie" is there since a long time, if you feel that this is too awkward for present times, maybe we could rebrand that into "magic header" or "Modules header" abbreviated "mheader" so to have a "mheader_check" option.
> I can change mcookie_check option name as 5.1 is not released yet. For mcookie_version_check I will need to wait for 6.0 (in 2 to 4 years) to change the name of this existing option.
> What is your opinion on that?
> Regards,
> Xavier
> 
>> Le sam. 15 janv. 2022 à 17:29, Paul Markfort <pau...@gm...> a écrit :
>> Maybe: module_marker_check
>> However, there is already an "mcookie_version_check" setting that is closely related (so mcookie_check seems quite appropriate).
>> Also, I should point out that mcookie is a contraction of Magic Cookie (that I have seen used in non module contexts - there is even an "mcookie" command in linux: mcookie - generate magic cookies for xauth).
>> 
>>> On 2022-01-15 07:57 AM, Sternberg, Michael G. via Modules-interest wrote:
>>> Xavier,
>>> This is a great development!  Dependency handling systems for HPC application deployment, like Spack, have made for some pretty deep and complex modulefile trees as of late.
>>> May I suggest a name change for the feature, though, if it isn't already committed?
>>> I ask because the term "cookie" could be misunderstood, as it has connotations with communication protocols, notably HTTP. Following convention, a more grokable name for the feature would be "[file ]magic" or "[file ]signature" – see  https://en.wikipedia.org/wiki/List_of_file_signatures .  The page even includes "#%Module" in its list of examples.
>>> Best wishes,
>>> --
>>> Michael Sternberg, Ph.D., Principal Scientific Computing Administrator
>>> Center for Nanoscale Materials, Argonne National Laboratory
>>> 
>>>> On Jan 14, 2022, at 00:43, Xavier Delaruelle <xav...@gm...> wrote:
>>>> Hello All,
>>>> Modules 5.1 work is ongoing. A new configuration option, mcookie_check, is introduced to skip the verification of each file in modulepath directories to determine if they are modulefiles or not.…

Re: [Modules] Overloading system modulerc by a user .modulerc file

[Sébastien G. <seb...@cc...>]

Dear Xavier,

Thanks a lot! This is exactly what we were looking for! We will indeed 
enable the switch to restore the default collection in our upcoming 
v5.0.1 deployment, and add some information on how to use it in our 
documentation.

Cheers,

Sébastien


Le 24/01/2022 à 20:24, Xavier Delaruelle a écrit :
> Dear Sébastien & Natthan,
>
> You may rely on collections for that: with the "module save" command, 
> users register their current environment which could be restored in 
> another session with "module restore" command. So in collections users 
> may set their specific modulepath entries and the module they want to 
> load in these modulepaths. More details on collections at [1]
>
> Then you may tune the global initrc file to restore the default 
> collection of the user if it exists. So when user logs in, its default 
> environment is automatically restored. See configuration step #3 in 
> installation guide [2].
>
> if  {[is-saved  default]}  {
>      module  restore
> }
>
> Cheers,
> Xavier
>
> [1] https://modules.readthedocs.io/en/latest/module.html#collections-1
> [2] https://modules.readthedocs.io/en/latest/INSTALL.html#configuration
>
> Le lun. 24 janv. 2022 à 18:36, <sup...@cc...> a écrit :
>
>     Dear Xavier,
>
>     Thanks for your explanations! It helps a lot, and we now
>     understand better how to use the initrc and rc files. However
>     there is still something we need to do which we don't know how to
>     achieve following your recommendations.
>
>>     Different kind of commands are found in each kind of files:
>>     * initrc: module config, module use, module load, module restore, ...
>>     * rc: module-alias, module-version, module-hide, module-tag,
>>     module-forbid, ...
>
>     We are providing Modules for all of our users. We would like them
>     to be able to add their own softwares using their own, customized
>     rc files. To do so, we first tried to use the 'module use' command
>     inside a rc file, but as you explained above, this is not the
>     right way to go.We initially did not understand that, and this is
>     what we believed to be an "unexpected behaviour". In fact, we can
>     use the 'module use' command inside a rc file, without any
>     explicite error, but it will overwrite the initrc configuration
>     (so we will lose our default Modules configuration).
>
>     So is there a way for usersto add a path to the MODULEPATH
>     variable using theirown rc file ?
>
>     Thanks & cheers,
>
>     Sébastien & Natthan
>     _______________________________________________
>     Modules-interest mailing list
>     Mod...@li...
>     https://lists.sourceforge.net/lists/listinfo/modules-interest
>
>
>
> _______________________________________________
> Modules-interest mailing list
> Mod...@li...
> https://lists.sourceforge.net/lists/listinfo/modules-interest

-- 
Sébastien Gadrat
CMS Collaboration @LHC/CERN - CMS Tier-1 Contact Person
CNRS/IN2P3 Computing Center
Lyon/France
Phone: +33 (0)4 78 93 08 80

Re: [Modules] module display issue

[Shaun D. <Sha...@us...>]

Hi Xavier,

Thank you for the explanation – I understand…

Best regards
Shaun

From: Xavier Delaruelle <xav...@gm...>
Sent: Tuesday, January 25, 2022 8:06 PM
To: Environment Modules usage and discussion. <mod...@li...>
Subject: Re: [Modules] module display issue

<snip>

Hi Shaun,

You are doing it right. The 'getenv' modulefile command has a specific behavior when modulefile is evaluated in display mode: it returns the environment variable name prefixed by the '$' sign.

So 'getenv' should not be used in conjunction of commands that requires the environment variable value if one want to display the modulefile without error.

On the next version (5.1), I have added a --return-value option to getenv [1] to tell this command to also return the environment variable value when evaluated in display mode.

Best regards,
Xavier

[1] https://modules.readthedocs.io/en/latest/MIGRATING.html#change-modulefile-command-behavior

Re: [Modules] module display issue

[Xavier D. <xav...@gm...>]

Le mar. 25 janv. 2022 à 23:13, Shaun Deacon <Sha...@us...>
a écrit :

> Hi,
>
>
>
> I hope it’s OK to post this here…
>
> I have an issue with “module display <modulename>” and use of “[getenv
> <environment>]” with shell script sourcing in the module (Modules version
> 4.7.1).
>
>
>
> Consider the following test module “display-test” :
>
>
>
> #%Module1.0
>
> #
>
> # Display test
>
> setenv CADROOT "/home/sdeacon"
>
> switch -- [module-info shelltype] {
>
>     sh {
>
>       puts stderr “Source echo.sh”
>
>       source-sh bash $env(CADROOT)/echo.sh
>
>     }
>
>     csh {
>
>       puts stderr “Source echo.csh”
>
>       source-sh tcsh $env(CADROOT)/echo.csh
>
>     }
>
>     default {}
>
> }
>
>
>
> Where “echo.sh” and “echo.csh” are simple bash and csh scripts for test
> purposes – they just echo “hello”…
>
> If I do “module display display-test” in a csh/tcsh, the output is as
> expected :
>
>
>
> -------------------------------------------------------------------
>
> /home/sdeacon/modules/display-test:
>
>
>
> setenv          CADROOT /home/sdeacon
>
> Source echo.csh
>
> -------------------------------------------------------------------
>
>
>
> However, if I modify the module in the following way :
>
>
>
> #%Module1.0
>
> #
>
> # Display test
>
> setenv CADROOT "/home/sdeacon"
>
> switch -- [module-info shelltype] {
>
>     sh {
>
>       puts stderr “Source echo.sh”
>
>       source-sh bash [getenv CADROOT]/echo.sh
>
>     }
>
>     csh {
>
>       puts stderr “Source echo.csh”
>
>       source-sh tcsh [getenv CADROOT]/echo.csh
>
>     }
>
>     default {}
>
> }
>
>
>
> I get the following issue with ‘module display’ :
>
>
>
> -------------------------------------------------------------------
>
> /home/sdeacon/modules/display-test:
>
>
>
> setenv          CADROOT /home/sdeacon
>
> Source echo.csh
>
> Module ERROR: Script '$CADROOT/echo.csh' cannot be found
>
>       while executing
>
>   "source-sh tcsh [getenv CADROOT]/echo.csh"
>
>       ("csh" arm line 3)
>
>       invoked from within
>
>   "switch -- [module-info shelltype] {
>
>       sh {
>
>           puts stderr "Source echo.sh"
>
>           source-sh bash [getenv CADROOT]/echo.sh
>
>       }
>
>       csh {
>
>       ..."
>
>       (file "/home/sdeacon/modules/display-test" line 5)
>
>   Please contact <root@localhost>
>
> -------------------------------------------------------------------
>
>
>
> Additionally, I also get the same kind of error if I do the following :
>
>
>
> #%Module1.0
>
> #
>
> # Display test
>
> setenv CADROOT "/home/sdeacon"
>
> setenv SCRIPT_HOME [getenv CADROOT]
>
> switch -- [module-info shelltype] {
>
>     sh {
>
>       puts stderr “Source echo.sh”
>
>       source-sh bash $env(SCRIPT_HOME)/echo.sh
>
>     }
>
>     csh {
>
>       puts stderr “Source echo.csh”
>
>       source-sh tcsh $env(SCRIPT_HOME)/echo.csh
>
>     }
>
>     default {}
>
> }
>
>
>
> Result of “module display” :
>
>
>
> -------------------------------------------------------------------
>
> /home/sdeacon/modules/display-test:
>
>
>
> setenv          CADROOT /home/sdeacon
>
> setenv          SCRIPT_HOME $CADROOT
>
> Source echo.csh
>
> Module ERROR: Script '$CADROOT/echo.csh' cannot be found
>
>       while executing
>
>   "source-sh tcsh $env(SCRIPT_HOME)/echo.csh"
>
>       ("csh" arm line 3)
>
>       invoked from within
>
>   "switch -- [module-info shelltype] {
>
>       sh {
>
>           puts stderr "Source echo.sh"
>
>           source-sh bash $env(SCRIPT_HOME)/echo.sh
>
>       }
>
>       csh {
>
>      ..."
>
>       (file "/home/sdeacon/modules/display-test" line 6)
>
>   Please contact <root@localhost>
>
> -------------------------------------------------------------------
>
>
>
> Am I doing something wrong ?
>
>
>
> Best regards,
>
> Shaun
>

Hi Shaun,

You are doing it right. The 'getenv' modulefile command has a specific
behavior when modulefile is evaluated in display mode: it returns the
environment variable name prefixed by the '$' sign.

So 'getenv' should not be used in conjunction of commands that requires the
environment variable value if one want to display the modulefile without
error.

On the next version (5.1), I have added a --return-value option to getenv
[1] to tell this command to also return the environment variable value when
evaluated in display mode.

Best regards,
Xavier

[1]
https://modules.readthedocs.io/en/latest/MIGRATING.html#change-modulefile-command-behavior

[Modules] module display issue

[Shaun D. <Sha...@us...>]

Hi,

I hope it's OK to post this here...
I have an issue with "module display <modulename>" and use of "[getenv <environment>]" with shell script sourcing in the module (Modules version 4.7.1).

Consider the following test module "display-test" :

#%Module1.0
#
# Display test
setenv CADROOT "/home/sdeacon"
switch -- [module-info shelltype] {
    sh {
      puts stderr "Source echo.sh"
      source-sh bash $env(CADROOT)/echo.sh
    }
    csh {
      puts stderr "Source echo.csh"
      source-sh tcsh $env(CADROOT)/echo.csh
    }
    default {}
}

Where "echo.sh" and "echo.csh" are simple bash and csh scripts for test purposes - they just echo "hello"...
If I do "module display display-test" in a csh/tcsh, the output is as expected :

-------------------------------------------------------------------
/home/sdeacon/modules/display-test:

setenv          CADROOT /home/sdeacon
Source echo.csh
-------------------------------------------------------------------

However, if I modify the module in the following way :

#%Module1.0
#
# Display test
setenv CADROOT "/home/sdeacon"
switch -- [module-info shelltype] {
    sh {
      puts stderr "Source echo.sh"
      source-sh bash [getenv CADROOT]/echo.sh
    }
    csh {
      puts stderr "Source echo.csh"
      source-sh tcsh [getenv CADROOT]/echo.csh
    }
    default {}
}

I get the following issue with 'module display' :

-------------------------------------------------------------------
/home/sdeacon/modules/display-test:

setenv          CADROOT /home/sdeacon
Source echo.csh
Module ERROR: Script '$CADROOT/echo.csh' cannot be found
      while executing
  "source-sh tcsh [getenv CADROOT]/echo.csh"
      ("csh" arm line 3)
      invoked from within
  "switch -- [module-info shelltype] {
      sh {
          puts stderr "Source echo.sh"
          source-sh bash [getenv CADROOT]/echo.sh
      }
      csh {
      ..."
      (file "/home/sdeacon/modules/display-test" line 5)
  Please contact <root@localhost>
-------------------------------------------------------------------

Additionally, I also get the same kind of error if I do the following :

#%Module1.0
#
# Display test
setenv CADROOT "/home/sdeacon"
setenv SCRIPT_HOME [getenv CADROOT]
switch -- [module-info shelltype] {
    sh {
      puts stderr "Source echo.sh"
      source-sh bash $env(SCRIPT_HOME)/echo.sh
    }
    csh {
      puts stderr "Source echo.csh"
      source-sh tcsh $env(SCRIPT_HOME)/echo.csh
    }
    default {}
}

Result of "module display" :

-------------------------------------------------------------------
/home/sdeacon/modules/display-test:

setenv          CADROOT /home/sdeacon
setenv          SCRIPT_HOME $CADROOT
Source echo.csh
Module ERROR: Script '$CADROOT/echo.csh' cannot be found
      while executing
  "source-sh tcsh $env(SCRIPT_HOME)/echo.csh"
      ("csh" arm line 3)
      invoked from within
  "switch -- [module-info shelltype] {
      sh {
          puts stderr "Source echo.sh"
          source-sh bash $env(SCRIPT_HOME)/echo.sh
      }
      csh {
     ..."
      (file "/home/sdeacon/modules/display-test" line 6)
  Please contact <root@localhost>
-------------------------------------------------------------------

Am I doing something wrong ?

Best regards,
Shaun

--
Shaun Deacon
Senior Staff Software Engineer
Socionext America, Inc.
phone: ++1 408 212 7410
--

Re: [Modules] Overloading system modulerc by a user .modulerc file

[Xavier D. <xav...@gm...>]

Dear Sébastien & Natthan,

You may rely on collections for that: with the "module save" command, users
register their current environment which could be restored in another
session with "module restore" command. So in collections users may set
their specific modulepath entries and the module they want to load in these
modulepaths. More details on collections at [1]

Then you may tune the global initrc file to restore the default collection
of the user if it exists. So when user logs in, its default environment is
automatically restored. See configuration step #3 in installation guide [2].

if {[is-saved default]} {
    module restore}


Cheers,
Xavier

[1] https://modules.readthedocs.io/en/latest/module.html#collections-1
[2] https://modules.readthedocs.io/en/latest/INSTALL.html#configuration

Le lun. 24 janv. 2022 à 18:36, <sup...@cc...> a écrit :

> Dear Xavier,
> Thanks for your explanations! It helps a lot, and we now understand better
> how to use the initrc and rc files. However there is still something we
> need to do which we don't know how to achieve following your
> recommendations.
>
> Different kind of commands are found in each kind of files:
> * initrc: module config, module use, module load, module restore, ...
> * rc: module-alias, module-version, module-hide, module-tag,
> module-forbid, ...
>
>
> We are providing Modules for all of our users. We would like them to be
> able to add their own softwares using their own, customized rc files. To
> do so, we first tried to use the 'module use' command inside a rc file,
> but as you explained above, this is not the right way to go. We initially
> did not understand that, and this is what we believed to be an "unexpected
> behaviour". In fact, we can use the 'module use' command inside a rc file,
> without any explicite error, but it will overwrite the initrc configuration
> (so we will lose our default Modules configuration).
>
> So is there a way for users to add a path to the MODULEPATH variable using
> their own rc file ?
>
> Thanks & cheers,
>
> Sébastien & Natthan
> _______________________________________________
> Modules-interest mailing list
> Mod...@li...
> https://lists.sourceforge.net/lists/listinfo/modules-interest
>

Re: [Modules] Overloading system modulerc by a user .modulerc file

[<sup...@cc...>]

Dear Xavier,

Thanks for your explanations! It helps a lot, and we now understand 
better how to use the initrc and rc files. However there is still 
something we need to do which we don't know how to achieve following 
your recommendations.

> Different kind of commands are found in each kind of files:
> * initrc: module config, module use, module load, module restore, ...
> * rc: module-alias, module-version, module-hide, module-tag, 
> module-forbid, ...

We are providing Modules for all of our users. We would like them to be 
able to add their own softwares using their own, customized rc files. To 
do so, we first tried to use the 'module use' command inside a rc file, 
but as you explained above, this is not the right way to go.We initially 
did not understand that, and this is what we believed to be an 
"unexpected behaviour". In fact, we can use the 'module use' command 
inside a rc file, without any explicite error, but it will overwrite the 
initrc configuration (so we will lose our default Modules configuration).

So is there a way for usersto add a path to the MODULEPATH variable 
using theirown rc file ?

Thanks & cheers,

Sébastien & Natthan

Re: [Modules] Overloading system modulerc by a user .modulerc file

[Xavier D. <xav...@gm...>]

Le jeu. 20 janv. 2022 à 21:18, <sup...@cc...> a écrit :

> Dear Paul,
>
> Thank you for your quick reply. Please see our answers below.
>
>
> Le 19/01/2022 à 19:06, Paul Markfort a écrit :
> > What you want is unclear.
> > Do you want the user's rc file to be read instead of the global one?
>
> No, we want the user's rc file configuration to be added to the default
> one.
>
>
> > In such a case, the user could just set the: MODULERCFILE variable (to
> > an empty path like /nonexistant/).
> >
> >
> > Below is an explanation of what should be the current behavior
> > (note - in all cases the rc files must be valid module files - that
> > is, begin with the module magic cookie line):
> >
> > And "man module" from 4.7.1 (again - your paths are probably different):
> >       Upon  invocation  of modulecmd.tcl module run-command files are
> > sourced
> >        in the following order:
> >
> >        1. Global  RC  file  as   specified   by   MODULERCFILE
> > variable   or
> >           /stage/site/modules/4.7.1/etc/rc.   If   MODULERCFILE
> > points  to  a
> >           directory, the modulerc file in this directory is used as
> > global  RC
> >           file.
> >
> >        2. User specific module RC file $HOME/.modulerc
>
> We observe that the user's '$HOME/.modulerc' (if not empty) completely
> replaces the initial '$MODULERCFILE' file configuration. We would rather
> want the configuration from this file to be added to the default
> configuration.
>
> I hope this makes things clearer.
>
> Best regards,
>
> Natthan
>
> >
> >        3. All .modulerc and .version files found during modulefile
> > seeking.
> >
> >
> >
> >
> > On 2022-01-19 10:35 AM, sup...@cc... wrote:
> >> Dear Modules guys,
> >>
> >> We have installed Modules 4.7.1 on our compute cluster, and we
> >> provide it to our users.
> >>
> >> We have set up a specific configuration using a system 'initrc' (i.e.
> >> the '/etc/environment-modules/initrc').
> >>
> >> We have found out that this initrc configuration can be fully
> >> overwritten by a user '$HOME/.modulerc' if it exists. We would rather
> >> want the user's configuration (i.e. '.modulerc') to be added to the
> >> default existing configuration, i.e. from the 'initrc' file.
> >>
> >> How to ensure that the user's configuration overloads the default one?
> >>
> >> Thank you in advance,
> >>
> >> Best regards,
> >>
> >> Sébastien
>

Dear Sebastien, Natthan and CC team

Thanks for your messages. Could you provide more details on the observation
you made with a small example for instance.

There is 2 separate kind of configuration:

* the 'initrc' file, which is the configuration for the module command and
user environment run once when module is initialized
* the rc files that contribute to the definition and properties of the
available modulefiles and are evaluated each time a module command is
executed

Different kind of commands are found in each kind of files:
* initrc: module config, module use, module load, module restore, ...
* rc: module-alias, module-version, module-hide, module-tag, module-forbid,
...

>From what I can test, the rc files are correctly cascaded:

$ cat /etc/environment-modules/rc
#%Module
module-alias alfoo dot
$ cat $HOME/.modulerc
#%Module
module-alias albar dot
$ module avail -t al
global/user modulerc:
albar
alfoo

So a small example would help us to better understand what you want to
achieve and spot the issue.

Best regards,
Xavier

Re: [Modules] Overloading system modulerc by a user .modulerc file

[<sup...@cc...>]

Dear Paul,

Thank you for your quick reply. Please see our answers below.


Le 19/01/2022 à 19:06, Paul Markfort a écrit :
> What you want is unclear.
> Do you want the user's rc file to be read instead of the global one?

No, we want the user's rc file configuration to be added to the default one.


> In such a case, the user could just set the: MODULERCFILE variable (to 
> an empty path like /nonexistant/).
>
>
> Below is an explanation of what should be the current behavior
> (note - in all cases the rc files must be valid module files - that 
> is, begin with the module magic cookie line):
>
> And "man module" from 4.7.1 (again - your paths are probably different):
>       Upon  invocation  of modulecmd.tcl module run-command files are 
> sourced
>        in the following order:
>
>        1. Global  RC  file  as   specified   by   MODULERCFILE 
> variable   or
>           /stage/site/modules/4.7.1/etc/rc.   If   MODULERCFILE 
> points  to  a
>           directory, the modulerc file in this directory is used as 
> global  RC
>           file.
>
>        2. User specific module RC file $HOME/.modulerc

We observe that the user's '$HOME/.modulerc' (if not empty) completely 
replaces the initial '$MODULERCFILE' file configuration. We would rather 
want the configuration from this file to be added to the default 
configuration.

I hope this makes things clearer.

Best regards,

Natthan

>
>        3. All .modulerc and .version files found during modulefile 
> seeking.
>
>
>
>
> On 2022-01-19 10:35 AM, sup...@cc... wrote:
>> Dear Modules guys,
>>
>> We have installed Modules 4.7.1 on our compute cluster, and we 
>> provide it to our users.
>>
>> We have set up a specific configuration using a system 'initrc' (i.e. 
>> the '/etc/environment-modules/initrc').
>>
>> We have found out that this initrc configuration can be fully 
>> overwritten by a user '$HOME/.modulerc' if it exists. We would rather 
>> want the user's configuration (i.e. '.modulerc') to be added to the 
>> default existing configuration, i.e. from the 'initrc' file.
>>
>> How to ensure that the user's configuration overloads the default one?
>>
>> Thank you in advance,
>>
>> Best regards,
>>
>> Sébastien
>>
>>
>>
>>
>> _______________________________________________
>> Modules-interest mailing list
>> Mod...@li...
>> https://lists.sourceforge.net/lists/listinfo/modules-interest
>

Re: [Modules] Overloading system modulerc by a user .modulerc file

[Paul M. <pau...@gm...>]

What you want is unclear.
Do you want the user's rc file to be read instead of the global one?
In such a case, the user could just set the: MODULERCFILE variable (to an empty path like /nonexistant/).


Below is an explanation of what should be the current behavior
(note - in all cases the rc files must be valid module files - that is, begin with the module magic cookie line):

From: man module (for version 5.0.1 - paths may be different than in your setup - read your "man module"):

Upon  invocation  of modulecmd.tcl module run-command files are sourced
        in the following order:

        1. Global  RC  file  as   specified   by   MODULERCFILE   variable   or
           /stage/site/modules/5.0.1/etc/rc. If MODULERCFILE points to a direc-
           tory, the modulerc file in this directory is used as global RC file.

        2. User specific module RC file $HOME/.modulerc

        3. All .modulerc and .version files found during modulefile seeking.


And "man module" from 4.7.1 (again - your paths are probably different):
       Upon  invocation  of modulecmd.tcl module run-command files are sourced
        in the following order:

        1. Global  RC  file  as   specified   by   MODULERCFILE   variable   or
           /stage/site/modules/4.7.1/etc/rc.   If   MODULERCFILE  points  to  a
           directory, the modulerc file in this directory is used as global  RC
           file.

        2. User specific module RC file $HOME/.modulerc

        3. All .modulerc and .version files found during modulefile seeking.




On 2022-01-19 10:35 AM, sup...@cc... wrote:
> Dear Modules guys,
> 
> We have installed Modules 4.7.1 on our compute cluster, and we provide it to our users.
> 
> We have set up a specific configuration using a system 'initrc' (i.e. the '/etc/environment-modules/initrc').
> 
> We have found out that this initrc configuration can be fully overwritten by a user '$HOME/.modulerc' if it exists. We would rather want the user's configuration (i.e. '.modulerc') to be added to the default existing configuration, i.e. from the 'initrc' file.
> 
> How to ensure that the user's configuration overloads the default one?
> 
> Thank you in advance,
> 
> Best regards,
> 
> Sébastien
> 
> 
> 
> 
> _______________________________________________
> Modules-interest mailing list
> Mod...@li...
> https://lists.sourceforge.net/lists/listinfo/modules-interest

-- 
--------------------------------------------------------
The views and opinions expressed above are strictly
those of the author(s).  The content of this message has
not been reviewed nor approved by any entity whatsoever.
--------------------------------------------------------
Paul FM         Info: http://paulfm.com/~paulfm/
--------------------------------------------------------

[Modules] Overloading system modulerc by a user .modulerc file

[<sup...@cc...>]

Dear Modules guys,

We have installed Modules 4.7.1 on our compute cluster, and we provide 
it to our users.

We have set up a specific configuration using a system 'initrc' (i.e. 
the '/etc/environment-modules/initrc').

We have found out that this initrc configuration can be fully 
overwritten by a user '$HOME/.modulerc' if it exists. We would rather 
want the user's configuration (i.e. '.modulerc') to be added to the 
default existing configuration, i.e. from the 'initrc' file.

How to ensure that the user's configuration overloads the default one?

Thank you in advance,

Best regards,

Sébastien

[Modules] Modules @Easybuild User Meeting 2022

[Xavier D. <xav...@gm...>]

Dear All,

I will do a presentation of the recent Modules activities on January 27th
at 14:00 UTC during the 7th Easybuild User Meeting (held online this year).

https://easybuild.io/eum22/

I highly recommend you this event that takes place from Monday 24th to
Friday 27th. Like every year, there will be a large spectrum of
presentation around the HPC application field.

Regards,
Xavier

Re: [Modules] Reducing number of I/O operations

[Xavier D. <xav...@gm...>]

This terminology "Modules magic cookie" is there since a long time, if you
feel that this is too awkward for present times, maybe we could rebrand
that into "magic header" or "Modules header" abbreviated "mheader" so to
have a "mheader_check" option.

I can change mcookie_check option name as 5.1 is not released yet. For
mcookie_version_check I will need to wait for 6.0 (in 2 to 4 years) to
change the name of this existing option.

What is your opinion on that?

Regards,
Xavier

Le sam. 15 janv. 2022 à 17:29, Paul Markfort <pau...@gm...> a écrit :

>
> Maybe: module_marker_check
>
> However, there is already an "mcookie_version_check" setting that is
> closely related (so mcookie_check seems quite appropriate).
>
> Also, I should point out that mcookie is a contraction of Magic Cookie
> (that I have seen used in non module contexts - there is even an "mcookie"
> command in linux: mcookie - generate magic cookies for xauth).
>
>
> On 2022-01-15 07:57 AM, Sternberg, Michael G. via Modules-interest wrote:
> > Xavier,
> >
> > This is a great development!  Dependency handling systems for HPC
> application deployment, like Spack, have made for some pretty deep and
> complex modulefile trees as of late.
> >
> > May I suggest a name change for the feature, though, if it isn't already
> committed?
> >
> > I ask because the term "cookie" could be misunderstood, as it has
> connotations with communication protocols, notably HTTP. Following
> convention, a more grokable name for the feature would be "[file ]magic" or
> "[file ]signature" – see
> https://en.wikipedia.org/wiki/List_of_file_signatures .  The page even
> includes "#%Module" in its list of examples.
> >
> >
> > Best wishes,
> > --
> > Michael Sternberg, Ph.D., Principal Scientific Computing Administrator
> > Center for Nanoscale Materials, Argonne National Laboratory
> >
> >
> > On Jan 14, 2022, at 00:43, Xavier Delaruelle <
> xav...@gm...> wrote:
> > Hello All,
> >
> > Modules 5.1 work is ongoing. A new configuration option, mcookie_check,
> is introduced to skip the verification of each file in modulepath
> directories to determine if they are modulefiles or not. When this option
> is set to `eval`, file checks only occur when an attempt is made to
> evaluate these files, not when listing available modules. So if you fully
> control what files are saved under the modulepath directories you could get
> a nice speed improvement, especially for commands like 'module avail':
> >
> >
> https://modules.readthedocs.io/en/latest/MIGRATING.html#reducing-number-of-i-o-operations
> >
> > A full cookbook recipe is also available to describe the features of
> Modules that help to reduce the I/O load of walking through the available
> modulefiles:
> >
> >
> https://modules.readthedocs.io/en/latest/cookbook/reduce-io-load.html#reduce-io-load
> >
> > Regards,
> > Xavier
> >
> >
> >
> > _______________________________________________
> > Modules-interest mailing list
> > Mod...@li...
> > https://lists.sourceforge.net/lists/listinfo/modules-interest
> >
>
> --
> --------------------------------------------------------
> The views and opinions expressed above are strictly
> those of the author(s).  The content of this message has
> not been reviewed nor approved by any entity whatsoever.
> --------------------------------------------------------
> Paul FM         Info: http://paulfm.com/~paulfm/
> --------------------------------------------------------
>
>
> _______________________________________________
> Modules-interest mailing list
> Mod...@li...
> https://lists.sourceforge.net/lists/listinfo/modules-interest
>

Re: [Modules] Reducing number of I/O operations

[Paul M. <pau...@gm...>]

Maybe: module_marker_check

However, there is already an "mcookie_version_check" setting that is closely related (so mcookie_check seems quite appropriate).

Also, I should point out that mcookie is a contraction of Magic Cookie (that I have seen used in non module contexts - there is even an "mcookie" command in linux: mcookie - generate magic cookies for xauth).


On 2022-01-15 07:57 AM, Sternberg, Michael G. via Modules-interest wrote:
> Xavier,
> 
> This is a great development!  Dependency handling systems for HPC application deployment, like Spack, have made for some pretty deep and complex modulefile trees as of late.
> 
> May I suggest a name change for the feature, though, if it isn't already committed?
> 
> I ask because the term "cookie" could be misunderstood, as it has connotations with communication protocols, notably HTTP. Following convention, a more grokable name for the feature would be "[file ]magic" or "[file ]signature" – see  https://en.wikipedia.org/wiki/List_of_file_signatures .  The page even includes "#%Module" in its list of examples.
> 
> 
> Best wishes,
> --
> Michael Sternberg, Ph.D., Principal Scientific Computing Administrator
> Center for Nanoscale Materials, Argonne National Laboratory
> 
> 
> On Jan 14, 2022, at 00:43, Xavier Delaruelle <xav...@gm...> wrote:
> Hello All,
> 
> Modules 5.1 work is ongoing. A new configuration option, mcookie_check, is introduced to skip the verification of each file in modulepath directories to determine if they are modulefiles or not. When this option is set to `eval`, file checks only occur when an attempt is made to evaluate these files, not when listing available modules. So if you fully control what files are saved under the modulepath directories you could get a nice speed improvement, especially for commands like 'module avail':
> 
> https://modules.readthedocs.io/en/latest/MIGRATING.html#reducing-number-of-i-o-operations
> 
> A full cookbook recipe is also available to describe the features of Modules that help to reduce the I/O load of walking through the available modulefiles:
> 
> https://modules.readthedocs.io/en/latest/cookbook/reduce-io-load.html#reduce-io-load
> 
> Regards,
> Xavier
> 
> 
> 
> _______________________________________________
> Modules-interest mailing list
> Mod...@li...
> https://lists.sourceforge.net/lists/listinfo/modules-interest
> 

-- 
--------------------------------------------------------
The views and opinions expressed above are strictly
those of the author(s).  The content of this message has
not been reviewed nor approved by any entity whatsoever.
--------------------------------------------------------
Paul FM         Info: http://paulfm.com/~paulfm/
--------------------------------------------------------

Re: [Modules] Reducing number of I/O operations

[Sternberg, M. G. <ste...@an...>]

Xavier,

This is a great development!  Dependency handling systems for HPC application deployment, like Spack, have made for some pretty deep and complex modulefile trees as of late.

May I suggest a name change for the feature, though, if it isn't already committed?

I ask because the term "cookie" could be misunderstood, as it has connotations with communication protocols, notably HTTP. Following convention, a more grokable name for the feature would be "[file ]magic" or "[file ]signature" – see  https://en.wikipedia.org/wiki/List_of_file_signatures .  The page even includes "#%Module" in its list of examples.


Best wishes,
--
Michael Sternberg, Ph.D., Principal Scientific Computing Administrator
Center for Nanoscale Materials, Argonne National Laboratory


On Jan 14, 2022, at 00:43, Xavier Delaruelle <xav...@gm...> wrote:
Hello All,

Modules 5.1 work is ongoing. A new configuration option, mcookie_check, is introduced to skip the verification of each file in modulepath directories to determine if they are modulefiles or not. When this option is set to `eval`, file checks only occur when an attempt is made to evaluate these files, not when listing available modules. So if you fully control what files are saved under the modulepath directories you could get a nice speed improvement, especially for commands like 'module avail':

https://modules.readthedocs.io/en/latest/MIGRATING.html#reducing-number-of-i-o-operations

A full cookbook recipe is also available to describe the features of Modules that help to reduce the I/O load of walking through the available modulefiles:

https://modules.readthedocs.io/en/latest/cookbook/reduce-io-load.html#reduce-io-load

Regards,
Xavier

Re: [Modules] Reducing number of I/O operations

[Xavier D. <xav...@gm...>]

Hi Paul,

The configuration option should be looked at rather its corresponding
superseding environment variable. Module configuration option has default
value and can then be superseded by setting a MODULE_* environment
variable. Configuration option handling can be controlled with "module
config" command. By default it lists all known configuration options:

$ module config
Modules Release 5.0.1+221-g754913fe (2022-01-10)

- Config. name ---------.- Value (set by if default overridden)
---------------
advanced_version_spec     1
auto_handling             1 (env-var)
avail_indepth             1
...

When using the default value set when Modules is built (through the
./configure script), module config will just report this value:

$ module config mcookie_check
Modules Release 5.0.1+221-g754913fe (2022-01-10)

- Config. name ---------.- Value (set by if default overridden)
---------------
mcookie_check             always

You can also use this command to set a specific value to the config option
(it will set the corresponding MODULES_* environment variable):

$ module config mcookie_check eval

When the default value is superseded, a '(env-var)' mention appear:

$ module config mcookie_check
Modules Release 5.0.1+221-g754913fe (2022-01-10)

- Config. name ---------.- Value (set by if default overridden)
---------------
mcookie_check             eval (env-var)

The initrc file, evaluated when the module command initializes in user's
shell session, can host "module config" commands. This is the good location
to define your site-specific configuration. See the configuration section
of the installation guide:

https://modules.readthedocs.io/en/latest/INSTALL.html#configuration

I will look at better explaining the default value of configuration option
in the documentation.

Best regards,
Xavier



Le ven. 14 janv. 2022 à 17:37, Paul Markfort <pau...@gm...> a écrit :

> Note:
>         module config --dump-state
>
> only lists the environment variables that are set. Unless there is another
> command that does this, maybe add a "module config -dump-state-all" that
> lists all the environment variables that could be set and the current
> assumed values (or better, or additionally, a -dump-state-unset to only
> list those that are not set).
>
>
> On 2022-01-14 10:23 AM, Paul Markfort wrote:
> > Great to see this improvement.
> >
> > Do you have a reference on how to change the assumed default values for
> any of the MODULES_* environment variables (the value assumed if the
> environment variable is not set) in the siteconfig.tcl file?  In
> particular, I would want to have MODULES_MCOOKIE_CHECK assumed to be eval
> (for the site, not in the distribution); but, there are others that might
> be worthwhile to have a site default for as well.
> >
> > Is there a module command that can list all these environment variables
> and what the current (assumed or set) setting is for each?  Perhaps that
> command (if it exists) should be listed on the documentation pages just
> below the ENVIRONMENT heading somehow.
> >
> >
> > Also, in the documentation of the environment variables, the default
> assumed values for many of them is not listed.  May I recommend that the
> default value be listed right after the variable either like this:
> > MODULES_MCOOKIE_CHECK        default_value: always
> >
> > or:
> > MODULES_MCOOKIE_CHECK
> >          default_value: always
> >
> >
> > On 2022-01-14 12:41 AM, Xavier Delaruelle wrote:
> >> Hello All,
> >>
> >> Modules 5.1 work is ongoing. A new configuration option, mcookie_check,
> is
> >> introduced to skip the verification of each file in modulepath
> directories
> >> to determine if they are modulefiles or not. When this option is set to
> >> `eval`, file checks only occur when an attempt is made to evaluate these
> >> files, not when listing available modules. So if you fully control what
> >> files are saved under the modulepath directories you could get a nice
> speed
> >> improvement, especially for commands like 'module avail':
> >>
> >>
> https://modules.readthedocs.io/en/latest/MIGRATING.html#reducing-number-of-i-o-operations
> >>
> >> A full cookbook recipe is also available to describe the features of
> >> Modules that help to reduce the I/O load of walking through the
> available
> >> modulefiles:
> >>
> >>
> https://modules.readthedocs.io/en/latest/cookbook/reduce-io-load.html#reduce-io-load
> >>
> >> Regards,
> >> Xavier
> >>
> >>
> >>
> >> _______________________________________________
> >> Modules-interest mailing list
> >> Mod...@li...
> >> https://lists.sourceforge.net/lists/listinfo/modules-interest
> >>
> >
>
> --
> --------------------------------------------------------
> The views and opinions expressed above are strictly
> those of the author(s).  The content of this message has
> not been reviewed nor approved by any entity whatsoever.
> --------------------------------------------------------
> Paul FM         Info: http://paulfm.com/~paulfm/
> --------------------------------------------------------
>
>
> _______________________________________________
> Modules-interest mailing list
> Mod...@li...
> https://lists.sourceforge.net/lists/listinfo/modules-interest
>

Re: [Modules] Reducing number of I/O operations

[Paul M. <pau...@gm...>]

Note:
	module config --dump-state

only lists the environment variables that are set. Unless there is another command that does this, maybe add a "module config -dump-state-all" that lists all the environment variables that could be set and the current assumed values (or better, or additionally, a -dump-state-unset to only list those that are not set).


On 2022-01-14 10:23 AM, Paul Markfort wrote:
> Great to see this improvement.
> 
> Do you have a reference on how to change the assumed default values for any of the MODULES_* environment variables (the value assumed if the environment variable is not set) in the siteconfig.tcl file?  In particular, I would want to have MODULES_MCOOKIE_CHECK assumed to be eval (for the site, not in the distribution); but, there are others that might be worthwhile to have a site default for as well.
> 
> Is there a module command that can list all these environment variables and what the current (assumed or set) setting is for each?  Perhaps that command (if it exists) should be listed on the documentation pages just below the ENVIRONMENT heading somehow.
> 
> 
> Also, in the documentation of the environment variables, the default assumed values for many of them is not listed.  May I recommend that the default value be listed right after the variable either like this:
> MODULES_MCOOKIE_CHECK        default_value: always
> 
> or:
> MODULES_MCOOKIE_CHECK
>          default_value: always
> 
> 
> On 2022-01-14 12:41 AM, Xavier Delaruelle wrote:
>> Hello All,
>>
>> Modules 5.1 work is ongoing. A new configuration option, mcookie_check, is
>> introduced to skip the verification of each file in modulepath directories
>> to determine if they are modulefiles or not. When this option is set to
>> `eval`, file checks only occur when an attempt is made to evaluate these
>> files, not when listing available modules. So if you fully control what
>> files are saved under the modulepath directories you could get a nice speed
>> improvement, especially for commands like 'module avail':
>>
>> https://modules.readthedocs.io/en/latest/MIGRATING.html#reducing-number-of-i-o-operations
>>
>> A full cookbook recipe is also available to describe the features of
>> Modules that help to reduce the I/O load of walking through the available
>> modulefiles:
>>
>> https://modules.readthedocs.io/en/latest/cookbook/reduce-io-load.html#reduce-io-load
>>
>> Regards,
>> Xavier
>>
>>
>>
>> _______________________________________________
>> Modules-interest mailing list
>> Mod...@li...
>> https://lists.sourceforge.net/lists/listinfo/modules-interest
>>
> 

-- 
--------------------------------------------------------
The views and opinions expressed above are strictly
those of the author(s).  The content of this message has
not been reviewed nor approved by any entity whatsoever.
--------------------------------------------------------
Paul FM         Info: http://paulfm.com/~paulfm/
--------------------------------------------------------

Re: [Modules] Reducing number of I/O operations

[Paul M. <pau...@gm...>]

Great to see this improvement.

Do you have a reference on how to change the assumed default values for any of the MODULES_* environment variables (the value assumed if the environment variable is not set) in the siteconfig.tcl file?  In particular, I would want to have MODULES_MCOOKIE_CHECK assumed to be eval (for the site, not in the distribution); but, there are others that might be worthwhile to have a site default for as well.

Is there a module command that can list all these environment variables and what the current (assumed or set) setting is for each?  Perhaps that command (if it exists) should be listed on the documentation pages just below the ENVIRONMENT heading somehow.


Also, in the documentation of the environment variables, the default assumed values for many of them is not listed.  May I recommend that the default value be listed right after the variable either like this:
MODULES_MCOOKIE_CHECK		default_value: always

or:
MODULES_MCOOKIE_CHECK
		default_value: always


On 2022-01-14 12:41 AM, Xavier Delaruelle wrote:
> Hello All,
> 
> Modules 5.1 work is ongoing. A new configuration option, mcookie_check, is
> introduced to skip the verification of each file in modulepath directories
> to determine if they are modulefiles or not. When this option is set to
> `eval`, file checks only occur when an attempt is made to evaluate these
> files, not when listing available modules. So if you fully control what
> files are saved under the modulepath directories you could get a nice speed
> improvement, especially for commands like 'module avail':
> 
> https://modules.readthedocs.io/en/latest/MIGRATING.html#reducing-number-of-i-o-operations
> 
> A full cookbook recipe is also available to describe the features of
> Modules that help to reduce the I/O load of walking through the available
> modulefiles:
> 
> https://modules.readthedocs.io/en/latest/cookbook/reduce-io-load.html#reduce-io-load
> 
> Regards,
> Xavier
> 
> 
> 
> _______________________________________________
> Modules-interest mailing list
> Mod...@li...
> https://lists.sourceforge.net/lists/listinfo/modules-interest
> 

-- 
--------------------------------------------------------
The views and opinions expressed above are strictly
those of the author(s).  The content of this message has
not been reviewed nor approved by any entity whatsoever.
--------------------------------------------------------
Paul FM         Info: http://paulfm.com/~paulfm/
--------------------------------------------------------