modules-interest — Discussion forum for modules usage.

[Modules] Forbidding use of modules

[Xavier D. <xav...@gm...>]

Dear All,

The `module-forbid` modulefile command is added to dynamically forbid the
evaluation of modulefiles it specifies. When forbidden, a module cannot be
loaded and an access error is returned when an attempt is made to evaluate
it.

    $ cat /path/to/modulefiles/foo/.modulerc
    #%Module4.6
    module-forbid foo@1:
    $ ml foo/1.0
    ERROR: Access to module 'foo/1.0' is denied
    $ ml
    No Modulefiles Currently Loaded.

`module-forbid` statements can be coupled with :mfcmd:`module-hide`
statements to hide modules in addition to forbid their use. `module-forbid`
supports the `--not-user`, `--not-group`, `--before` and `--after` options
to still allow some users or forbid modules before or after a given date
time.

An additional error message can be defined with the `--message` option to
guide for instance users when they try to evaluate a forbidden module:

    $ cat /path/to/modulefiles/bar/.modulerc
    #%Module4.6
    module-forbid --message {Software bar/1.0 is decommissioned, please now
use\
        bar/2.0} --after 2020-09-01 bar/1.0
    $ ml bar/1.0
    ERROR: Access to module 'bar/1.0' is denied
      Software bar/1.0 is decommissioned, please now use bar/2.0

When an evaluated module will soon be forbidden, a message is returned to
the user to warn him/her of the near limit. An additional warning message
can also be defined here with the `--nearly-message` option to guide users.

    $ cat /path/to/modulefiles/qux/.modulerc
    #%Module4.6
    module-forbid --nearly-message {Version 1.0 will soon expire, please
now use\
        version 2.0} --after 2020-09-15 qux/1.0
    $ date
    Tue 08 Sep 2020 06:49:43 AM CEST
    $ ml qux/1.0
    Loading qux/1.0
      WARNING: Access to module will be denied starting '2020-09-15'
        Version 1.0 will soon expire, please now use version 2.0

The range of time the *nearly forbidden* warning appears can be controlled
with the `nearly_forbidden_days` configuration option, whose value equals
to the number of days prior the module starts to be forbidden. This
configuration is set to `14` (days) by default and this value can be
controlled at `configure` time with `--with-nearly-forbidden-days` option.
When the `nearly_forbidden_days` configuration is set through the `config`
sub-command, the `MODULES_NEARLY_FORBIDDEN_DAYS` environment variable is
set.

This new feature is available in git repository and will be included in
next release (v4.6).

Regards,
Xavier

[Modules] Hiding modules

[Xavier D. <xav...@gm...>]

Dear All,

The newly introduced `module-hide` modulefile command enables to
dynamically hide modulefiles, module aliases or symbolic versions specified
to it:

    $ cat /path/to/modulefiles/bar/.modulerc
    #%Module4.6
    module-version bar/1.0 old
    # hide 'old' symbolic version
    module-hide bar/old
    # hide all version 2 and above
    module-hide bar@2:
    $ cat /path/to/modulefiles/.modulerc
    #%Module4.6
    # hide all versions of foo module
    module-hide foo

`module-hide` commands should be placed in module rc files and can leverage
the [Advanced module version specifiers](
https://timeline.noticeable.io/cbzeG7wTvAIqj21zbUmx/posts/advanced-module-version-specifiers)
syntax as shown in the above example.

Hidden modules are excluded from available module search or module
selection unless query refers to hidden module by its exact name:

    $ ml av
    --------------- /path/to/modulefiles ---------------
    bar/1.0  bar/2.0
    $ module load -v foo
    ERROR: Unable to locate a modulefile for 'foo'
    $ module load -v foo/1.0
    Loading foo/1.0
    $ module avail bar/old
    --------------- /path/to/modulefiles ---------------
    bar/1.0(old)

`module-hide` command accepts a `--soft` option to apply a lighter of
hiding to modules:

    $ cat /path/to/modulefiles/qux/.modulerc
    #%Module4.6
    # softly hide all qux modules
    module-hide --soft qux

The soft hiding mode enables to hide modules from full availability listing
yet keeping the ability to select such module for load without having to
use module exact name:

    $ ml av
    --------------- /path/to/modulefiles ---------------
    bar/1.0  bar/2.0
    $ ml av qux
    --------------- /path/to/modulefiles ---------------
    qux/1.0  qux/2.0
    $ module load -v qux
    Loading qux/2.0

Alternatively, a `--hard` option can be set on `module-hide` command to
ensure designated modules do not unveil even if referred by their exact
name:

    $ cat /path/to/modulefiles/qux/.modulerc
    #%Module4.6
    # softly hide all qux modules
    module-hide --soft qux
    # set highest version of qux hard hidden
    module-hide --hard qux/3.0
    $ ml av qux/3.0
    $ ml qux/3.0
    ERROR: Unable to locate a modulefile for 'qux/3.0'

Some users or groups can be set unaffected by hiding mechanism with the
`--not-user` or `--not-group` options:

    $ cat /path/to/modulefiles/quuz/.modulerc
    #%Module4.6
    # hiding does not apply to grp1 and grp2 groups
    module-hide --not-group {grp1 grp2} quuz

    $ id --groups --name
    grp1 grp7
    $ ml av quuz
    --------------- /path/to/modulefiles ---------------
    quuz/1.0  quuz/2.0
    $ ml -v quuz
    Loading quuz/2.0

Hiding mechanism can also be set effective only before or after a given
date time with the `--before` and `--after` options. Accepted date time
format is `YYYY-MM-DD[THH:MM]`.

    $ cat /path/to/modulefiles/fum/.modulerc
    #%Module4.6
    # hide only before a given date
    module-hide --hard --before 2020-09-01T12:00 fum/1.0
    # hide only after a given date
    module-hide --hard --after 2020-09-01 fum/2.0

    $ date
    Fri 04 Sep 2020 06:21:48 AM CEST
    $ ml av fum
    --------------- /path/to/modulefiles ---------------
    fum/1.0

Hidden modules can be included in available module searches if option
`--all`/`-a` is set on `avail`, `aliases`, `whatis` or `search`
sub-commands. Hard hidden modules are unaffected by this option and stay
hidden.

    $ ml av -a
    --------------- /path/to/modulefiles ---------------
    bar/1.0(old)  foo/1.0  fum/1.0   quuz/2.0  qux/2.0
    bar/2.0       foo/2.0  quuz/1.0  qux/1.0

This new feature is available in git repository and will be included in
next release (v4.6).

Regards,
Xavier

Re: [Modules] Modules 4.5.3 released

[Xavier D. <xav...@gm...>]

Hi Paul,

Such warning is already there in 4.5.3:

$ ./configure --enable-foo --with-bar=val
WARNING: Option `--enable-foo' ignored
WARNING: Option `--with-bar=val' ignored
...

Best regards,
Xavier

Le lun. 31 août 2020 à 22:25, Paul FM via Modules-interest <
mod...@li...> a écrit :

> And I just went through all the fun of getting 4.5.2 working just a couple
> weeks ago (so far no complaints).
>
> This "issue" (I would call it a feature):
> Install: ignore unsupported --enable-* and --with-* options on configure
> script rather raise an error and add support to define environment variable
> and build system type as configure script arguments to comply with GNU
> configuration recommendations.
>
> Got me to clean up some old configure options.
>
> It might be nice to print out a warning when configure is done about the
> ignored options.
>
>
> --  On 2020-08-31 13:54, Xavier Delaruelle wrote:
> > Hello All,
> >
> > Version 4.5.3 of Modules has just been released. It provides fixes for
> the
> > few issues recently spotted.
> >
> > The changes introduced in this bugfix release are detailed at:
> >
> >
> >
> https://timeline.noticeable.io/cbzeG7wTvAIqj21zbUmx/posts/modules-4-5-3-released
> >
> > The list of all changes made in this new version are described in the
> NEWS
> > document. Details on the new features introduced in the 4.5 series are
> > provided in the MIGRATING document.
> >
> >   http://modules.readthedocs.io/en/stable/MIGRATING.html
> >   http://modules.readthedocs.io/en/stable/NEWS.html
> >
> > The tarball of this new version can be downloaded at:
> >
> >   http://downloads.sourceforge.net/modules/modules-4.5.3.tar.gz
> >
> > The zipball to install this new version on Windows platform can be
> > downloaded at:
> >
> >   http://downloads.sourceforge.net/modules/modules-4.5.3-win.zip
> >
> > If you encounter any issue, please let us know by creating a ticket on
> the
> > project bug tracker at:
> >
> >   https://github.com/cea-hpc/modules/issues
> >
> > Special thanks to all the people who reported issues.
> >
> > Best regards,
> > Xavier
> >
> >
> >
> > _______________________________________________
> > Modules-interest mailing list
> > Mod...@li...
> > https://lists.sourceforge.net/lists/listinfo/modules-interest
> >
>
> --
> --------------------------------------------------------
> The views and opinions expressed above are strictly
> those of the author(s).  The content of this message has
> not been reviewed nor approved by any entity whatsoever.
> --------------------------------------------------------
> Paul FM         Info: http://umn.edu/~paulfm/
> --------------------------------------------------------
>
>
> _______________________________________________
> Modules-interest mailing list
> Mod...@li...
> https://lists.sourceforge.net/lists/listinfo/modules-interest
>

Re: [Modules] Modules 4.5.3 released

[Paul FM <pa...@um...>]

And I just went through all the fun of getting 4.5.2 working just a couple weeks ago (so far no complaints).

This "issue" (I would call it a feature):
Install: ignore unsupported --enable-* and --with-* options on configure script rather raise an error and add support to define environment variable and build system type as configure script arguments to comply with GNU configuration recommendations.

Got me to clean up some old configure options.

It might be nice to print out a warning when configure is done about the ignored options.


--  On 2020-08-31 13:54, Xavier Delaruelle wrote:
> Hello All,
> 
> Version 4.5.3 of Modules has just been released. It provides fixes for the
> few issues recently spotted.
> 
> The changes introduced in this bugfix release are detailed at:
> 
> 
> https://timeline.noticeable.io/cbzeG7wTvAIqj21zbUmx/posts/modules-4-5-3-released
> 
> The list of all changes made in this new version are described in the NEWS
> document. Details on the new features introduced in the 4.5 series are
> provided in the MIGRATING document.
> 
>   http://modules.readthedocs.io/en/stable/MIGRATING.html
>   http://modules.readthedocs.io/en/stable/NEWS.html
> 
> The tarball of this new version can be downloaded at:
> 
>   http://downloads.sourceforge.net/modules/modules-4.5.3.tar.gz
> 
> The zipball to install this new version on Windows platform can be
> downloaded at:
> 
>   http://downloads.sourceforge.net/modules/modules-4.5.3-win.zip
> 
> If you encounter any issue, please let us know by creating a ticket on the
> project bug tracker at:
> 
>   https://github.com/cea-hpc/modules/issues
> 
> Special thanks to all the people who reported issues.
> 
> Best regards,
> Xavier
> 
> 
> 
> _______________________________________________
> Modules-interest mailing list
> Mod...@li...
> https://lists.sourceforge.net/lists/listinfo/modules-interest
> 

-- 
--------------------------------------------------------
The views and opinions expressed above are strictly
those of the author(s).  The content of this message has
not been reviewed nor approved by any entity whatsoever.
--------------------------------------------------------
Paul FM         Info: http://umn.edu/~paulfm/
--------------------------------------------------------

[Modules] Modules 4.5.3 released

[Xavier D. <xav...@gm...>]

Hello All,

Version 4.5.3 of Modules has just been released. It provides fixes for the
few issues recently spotted.

The changes introduced in this bugfix release are detailed at:


https://timeline.noticeable.io/cbzeG7wTvAIqj21zbUmx/posts/modules-4-5-3-released

The list of all changes made in this new version are described in the NEWS
document. Details on the new features introduced in the 4.5 series are
provided in the MIGRATING document.

 http://modules.readthedocs.io/en/stable/MIGRATING.html
 http://modules.readthedocs.io/en/stable/NEWS.html

The tarball of this new version can be downloaded at:

 http://downloads.sourceforge.net/modules/modules-4.5.3.tar.gz

The zipball to install this new version on Windows platform can be
downloaded at:

 http://downloads.sourceforge.net/modules/modules-4.5.3-win.zip

If you encounter any issue, please let us know by creating a ticket on the
project bug tracker at:

 https://github.com/cea-hpc/modules/issues

Special thanks to all the people who reported issues.

Best regards,
Xavier

[Modules] Modules 4.5.2 released

[Xavier D. <xav...@gm...>]

Hello All,

Version 4.5.2 of Modules has just been released. It provides fixes for the
few issues recently spotted.

The changes introduced in this bugfix release are detailed at:

 https://timeline.noticeable.io/cbzeG7wTvAIqj21zbUmx/posts/modules-4-5-2
-released

The list of all changes made in this new version are described in the NEWS
document. Details on the new features introduced in the 4.5 series are
provided in the MIGRATING document.

 http://modules.readthedocs.io/en/stable/MIGRATING.html
 http://modules.readthedocs.io/en/stable/NEWS.html

The tarball of this new version can be downloaded at:

 http://downloads.sourceforge.net/modules/modules-4.5.2.tar.gz

The zipball to install this new version on Windows platform can be
downloaded at:

 http://downloads.sourceforge.net/modules/modules-4.5.2-win.zip

If you encounter any issue, please let us know by creating a ticket on the
project bug tracker at:

 https://github.com/cea-hpc/modules/issues

Special thanks to Rob Hurt for his code contribution on this release and to
all the people who reported issues.

Best regards,
Xavier

[Modules] Querying user's name and groups membership

[Xavier D. <xav...@gm...>]

Dear All,

Two new sub-commands are introduced for the `module-info` modulefile
command: `username` and `usergroups`. They respectively fetch the name of
the user currently running modulecmd.tcl or the name of all the groups this
user is member of.

These two new modulefile commands can help to adapt code to specific users
or groups. Like for instance to instantiate a modulefile for each group the
user is member of:

  $ cat /path/to/modulefiles/foo/.modulerc
  #%Module
  foreach grp [module-info usergroups] {
      module-virtual foo/$grp .common
  }
  $ id -G -n
  grp1 grp2 grp3
  $ module avail
  --------------- /path/to/modulefiles ---------------
  foo/grp1  foo/grp2  foo/grp3

`username` and `usergroups` sub-commands of `module-info` modulefile
command are only supported on Unix platform.

This new feature is available in git repository and will be included in
next release (v4.6).

Regards,
Xavier

[Modules] Embed shell scripts in modulefiles

[Xavier D. <xav...@gm...>]

Dear All,

The `source-sh` modulefile command is introduced to source environment
changes done by the evaluation of a shell script passed as argument. With
newly introduced `sh-to-mod` sub-command, resulting environment changes
done by script are output as modulefile commands. `source-sh` applies those
modulefile commands as if they were directly written in loading modulefile.

`source-sh` is useful for software providing a shell script for their
enablement. If you want to enable such software with `module` yet using
shell script provided by software for this task, just write a modulefile
using `source-sh` command to call the shell script.

Keeping the same example used to describe `sh-to-mod` sub-command: *foo*
software provides a `foo-setup.sh` script for its activation. Create a
modulefile `foo/1.2` that calls this script:

    $ cat /path/to/modulefiles/foo/1.2
    #%Module
    source-sh sh /path/to/foo-1.2/foo-setup.sh arg1

Displaying this modulefile indicates the environment changes done by script:

    $ module display foo/1.2
    -------------------------------------------------------------------
    /path/to/modulefiles/foo/1.2:

    prepend-path    PATH /path/to/foo-1.2/bin
    set-alias       foo {foobin -q -l}
    setenv          FOOENV arg1
    -------------------------------------------------------------------

Loading the modulefile applies the environment changes seen above:

    $ module load -v foo/1.2
    Loading foo/1.2
    $ echo $FOOENV
    arg1
    $ alias foo
    alias foo='foobin -q -l'

Track of these changes is kept in user environment to be able to undo them
when modulefile is unloaded:

    $ module unload -v foo/1.2
    Unloading foo/1.2
    $ echo $FOOENV

    $ alias foo
    bash: alias: foo: not found

Changes on environment variables, shell aliases, shell functions and
current working directory are tracked. The following shells are supported:
*sh*, *dash*, *csh*, *tcsh*, *bash*, *ksh*, *ksh93*, *zsh* and *fish*.

This new feature is available in git repository and will be included in
next release (v4.6).

Regards,
Xavier

Re: [Modules] module switch not preserving order

[Leo B. <leo...@gm...>]

Thanks, Xavier. This will be very useful in my env.

In my case our shell environment is more complicated than it should be
(working on that) and there are some order dependencies in the PATH
definition. I regularly do a "module switch" in order to test new versions
of EDA software or our toolchain. With v4 the only choice would be to
create a new meta-modulefile to select the desired versions and then load
that. It's of course doable, just not as flexible.

Regards,

- Leo

On Fri, Jun 26, 2020 at 12:23 PM Xavier Delaruelle <
xav...@gm...> wrote:

> I have made a ticket to track this need of getting back switch sub-command
> behavior of Modules version 3:
>
> https://github.com/cea-hpc/modules/issues/347
>
> Best regards,
> Xavier
>
> Le ven. 26 juin 2020 à 17:13, Roberto Beccherle <
> Rob...@pi...> a écrit :
>
>> I agree that the new behaviour is much better.
>> Nevertheless i understand that if you have many files that rely on the
>> old behaviour it would be best to have a default option (that can be set by
>> the user, or the modulefiles maintainer for all users using the modules)
>> that has compatibility with the "old" way of dealing with this.
>>
>> Roberto
>>
>> On Fri, Jun 26, 2020 at 2:54 PM Paul FM via Modules-interest <
>> mod...@li...> wrote:
>>
>>> You could add an option to switch/swap to choose this behavior.
>>>
>>> -o  (for preserve module order)
>>> -n  (for NO preserve module order)
>>> -i  (for do it the old inconsistent way)
>>>
>>> Then have something in the config that sets the default.
>>> (and maybe have module -h list the defaults).
>>>
>>> Personally, I like the new behavior (I was not even really aware of the
>>> old behavior).  The new behavior is more consistent with what I would
>>> expect.
>>>
>>> There are always going to be some modules which don't unload very well
>>> and if switch were to unload and reload those modules there could be side
>>> effects (like removing something from the environment that was set by
>>> something other than a module).
>>>
>>>
>>> --  On 2020-06-26 1:32 AM, Xavier Delaruelle wrote:
>>> > Hi Leo,
>>> >
>>> > What you encounter is the expected behavior starting version 4. I have
>>> > described this difference of behavior for the switch sub-command on the
>>> > diff doc [1].
>>> >
>>> > The switch in place of version 3 was very specific by replacing
>>> append-path
>>> > of prepend-path entries from the switched-on module at the exact same
>>> place
>>> > entries are removed for the switched-off module. It coped well when
>>> > switching modules which are really close but could lead to
>>> inconsistencies
>>> > when those 2 modules are different, like for instance if you want to
>>> switch
>>> > a compiler by another.
>>> >
>>> > One goal of version 4 is to bring consistency over the module actions,
>>> that
>>> > is why the behavior of the switch sub-command from version 3 has not
>>> been
>>> > ported to version 4.
>>> >
>>> > However it seems possible to have this behavior back if every loaded
>>> > modules after switched-off module are automatically unloaded, then
>>> > switched-on module is loaded and after that every automatically
>>> unloaded
>>> > modules are loaded again. This could be opt-in throught an option to
>>> have
>>> > switch behaves as "switch in place".
>>> >
>>> > Best regards,
>>> > Xavier
>>> >
>>> > [1]
>>> >
>>> https://modules.readthedocs.io/en/latest/diff_v3_v4.html#module-sub-commands
>>> >
>>> > Le jeu. 25 juin 2020 à 02:41, Leo Butler <leo...@gm...> a
>>> écrit :
>>> >
>>> >> Hi Xavier,
>>> >>
>>> >> I think perhaps this question has been asked and answered before but
>>> I've
>>> >> searched the archive and looked in MIGRATING.txt but haven't found
>>> it. If
>>> >> it's a duplicate, my apologies.
>>> >>
>>> >> Using the current version of modules (v4.5.1), loaded module order
>>> isn't
>>> >> preserved. To demonstrate, I set up four modulefiles that look like
>>> this:
>>> >>
>>> >> alpha/1: append-path TEST_VAR alpha1
>>> >> beta/2: append-path TEST_VAR beta2
>>> >> beta/4: append-path TEST_VAR beta4
>>> >> gamma/3: append-path TEST_VAR gamma3
>>> >>
>>> >> % module load alpha beta/2 gamma
>>> >> % module list
>>> >> Currently Loaded Modulefiles:
>>> >>   1) alpha/1   2) beta/2   3) gamma/3
>>> >> % echo $TEST_VAR
>>> >> alpha1:beta2:gamma3
>>> >> % module switch beta/4
>>> >> % module list
>>> >> Currently Loaded Modulefiles:
>>> >>   1) alpha/1   2) gamma/3   3) beta/4
>>> >> % echo $TEST_VAR
>>> >> alpha1:gamma3:beta4
>>> >>
>>> >> With v3.2, the original order is preserved, and I think this is the
>>> >> desired behavior.
>>> >>
>>> >> % module list
>>> >> Currently Loaded Modulefiles:
>>> >>    1) alpha/1   2) beta/4    3) gamma/3
>>> >> % echo $TEST_VAR
>>> >> alpha1:beta4:gamma3
>>> >>
>>> >> Anyone looking to move a modulefile into a different place in the
>>> order
>>> >> could simply do a load followed by an unload. The switch command
>>> seems to
>>> >> just be a proxy for this in v4.x, at least by default.
>>> >>
>>> >> I've looked for something to alter this behavior but I've not found
>>> it.
>>> >> Any suggestions?
>>> >>
>>> >> Thanks,
>>> >>
>>> >> - Leo
>>> >> _______________________________________________
>>> >> Modules-interest mailing list
>>> >> Mod...@li...
>>> >> https://lists.sourceforge.net/lists/listinfo/modules-interest
>>> >>
>>> >
>>> >
>>> >
>>> > _______________________________________________
>>> > Modules-interest mailing list
>>> > Mod...@li...
>>> > https://lists.sourceforge.net/lists/listinfo/modules-interest
>>> >
>>>
>>> --
>>> --------------------------------------------------------
>>> The views and opinions expressed above are strictly
>>> those of the author(s).  The content of this message has
>>> not been reviewed nor approved by any entity whatsoever.
>>> --------------------------------------------------------
>>> Paul FM         Info: http://umn.edu/~paulfm/
>>> --------------------------------------------------------
>>>
>>>
>>> _______________________________________________
>>> Modules-interest mailing list
>>> Mod...@li...
>>> https://lists.sourceforge.net/lists/listinfo/modules-interest
>>>
>> _______________________________________________
>> Modules-interest mailing list
>> Mod...@li...
>> https://lists.sourceforge.net/lists/listinfo/modules-interest
>>
> _______________________________________________
> Modules-interest mailing list
> Mod...@li...
> https://lists.sourceforge.net/lists/listinfo/modules-interest
>

[Modules] Translating shell scripts into modulefiles

[Xavier D. <xav...@gm...>]

Dear All,

The `sh-to-mod` sub-command is added to output as a modulefile content the
environment changes done by the evaluation of a shell script passed as
argument. `sh-to-mod` is especially useful for software providing a shell
script for their enablement in shell session: it can convert these scripts
into modulefiles.

Say for instance, a *foo* software has been installed and it provides a
`foo-setup.sh` script to activate *foo* software in user environment:

    $ cat /path/to/foo-1.2/foo-setup.sh
    #!/bin/sh
    export FOOENV="$1"
    export PATH=/path/to/foo-1.2/bin:$PATH
    alias foo='foobin -q -l'

Calling `module sh-to-mod` on this shell script outputs the environment
changes it performs as a modulefile content:

    $ module sh-to-mod sh /path/to/foo-1.2/foo-setup.sh arg1
    #%Module
    prepend-path    PATH /path/to/foo-1.2/bin
    set-alias       foo {foobin -q -l}
    setenv          FOOENV arg1

Changes on environment variables, shell aliases, shell functions and
current working directory are tracked. The following shells are supported:
*sh*, *dash*, *csh*, *tcsh*, *bash*, *ksh*, *ksh93*, *zsh* and *fish*.

`sh-to-mod` acts as a full replacement for the standalone `createmodule.sh`
and `createmodule.py` scripts. However those two scripts are currently
still provided for compatibility purpose.

This new feature is available in git repository and will be included in
next release (v4.6).

Regards,
Xavier

[Modules] @EnvModules, twitter feed for Modules project

[Xavier D. <xav...@gm...>]

Hello All,

Modules now has its own Twitter feed (@EnvModules):

https://twitter.com/EnvModules

It will be an additional way to get news and other announcements from the
project.

Cheers,
Xavier

Re: [Modules] module switch not preserving order

[Xavier D. <xav...@gm...>]

I have made a ticket to track this need of getting back switch sub-command
behavior of Modules version 3:

https://github.com/cea-hpc/modules/issues/347

Best regards,
Xavier

Le ven. 26 juin 2020 à 17:13, Roberto Beccherle <
Rob...@pi...> a écrit :

> I agree that the new behaviour is much better.
> Nevertheless i understand that if you have many files that rely on the old
> behaviour it would be best to have a default option (that can be set by the
> user, or the modulefiles maintainer for all users using the modules) that
> has compatibility with the "old" way of dealing with this.
>
> Roberto
>
> On Fri, Jun 26, 2020 at 2:54 PM Paul FM via Modules-interest <
> mod...@li...> wrote:
>
>> You could add an option to switch/swap to choose this behavior.
>>
>> -o  (for preserve module order)
>> -n  (for NO preserve module order)
>> -i  (for do it the old inconsistent way)
>>
>> Then have something in the config that sets the default.
>> (and maybe have module -h list the defaults).
>>
>> Personally, I like the new behavior (I was not even really aware of the
>> old behavior).  The new behavior is more consistent with what I would
>> expect.
>>
>> There are always going to be some modules which don't unload very well
>> and if switch were to unload and reload those modules there could be side
>> effects (like removing something from the environment that was set by
>> something other than a module).
>>
>>
>> --  On 2020-06-26 1:32 AM, Xavier Delaruelle wrote:
>> > Hi Leo,
>> >
>> > What you encounter is the expected behavior starting version 4. I have
>> > described this difference of behavior for the switch sub-command on the
>> > diff doc [1].
>> >
>> > The switch in place of version 3 was very specific by replacing
>> append-path
>> > of prepend-path entries from the switched-on module at the exact same
>> place
>> > entries are removed for the switched-off module. It coped well when
>> > switching modules which are really close but could lead to
>> inconsistencies
>> > when those 2 modules are different, like for instance if you want to
>> switch
>> > a compiler by another.
>> >
>> > One goal of version 4 is to bring consistency over the module actions,
>> that
>> > is why the behavior of the switch sub-command from version 3 has not
>> been
>> > ported to version 4.
>> >
>> > However it seems possible to have this behavior back if every loaded
>> > modules after switched-off module are automatically unloaded, then
>> > switched-on module is loaded and after that every automatically unloaded
>> > modules are loaded again. This could be opt-in throught an option to
>> have
>> > switch behaves as "switch in place".
>> >
>> > Best regards,
>> > Xavier
>> >
>> > [1]
>> >
>> https://modules.readthedocs.io/en/latest/diff_v3_v4.html#module-sub-commands
>> >
>> > Le jeu. 25 juin 2020 à 02:41, Leo Butler <leo...@gm...> a écrit
>> :
>> >
>> >> Hi Xavier,
>> >>
>> >> I think perhaps this question has been asked and answered before but
>> I've
>> >> searched the archive and looked in MIGRATING.txt but haven't found it.
>> If
>> >> it's a duplicate, my apologies.
>> >>
>> >> Using the current version of modules (v4.5.1), loaded module order
>> isn't
>> >> preserved. To demonstrate, I set up four modulefiles that look like
>> this:
>> >>
>> >> alpha/1: append-path TEST_VAR alpha1
>> >> beta/2: append-path TEST_VAR beta2
>> >> beta/4: append-path TEST_VAR beta4
>> >> gamma/3: append-path TEST_VAR gamma3
>> >>
>> >> % module load alpha beta/2 gamma
>> >> % module list
>> >> Currently Loaded Modulefiles:
>> >>   1) alpha/1   2) beta/2   3) gamma/3
>> >> % echo $TEST_VAR
>> >> alpha1:beta2:gamma3
>> >> % module switch beta/4
>> >> % module list
>> >> Currently Loaded Modulefiles:
>> >>   1) alpha/1   2) gamma/3   3) beta/4
>> >> % echo $TEST_VAR
>> >> alpha1:gamma3:beta4
>> >>
>> >> With v3.2, the original order is preserved, and I think this is the
>> >> desired behavior.
>> >>
>> >> % module list
>> >> Currently Loaded Modulefiles:
>> >>    1) alpha/1   2) beta/4    3) gamma/3
>> >> % echo $TEST_VAR
>> >> alpha1:beta4:gamma3
>> >>
>> >> Anyone looking to move a modulefile into a different place in the order
>> >> could simply do a load followed by an unload. The switch command seems
>> to
>> >> just be a proxy for this in v4.x, at least by default.
>> >>
>> >> I've looked for something to alter this behavior but I've not found it.
>> >> Any suggestions?
>> >>
>> >> Thanks,
>> >>
>> >> - Leo
>> >> _______________________________________________
>> >> Modules-interest mailing list
>> >> Mod...@li...
>> >> https://lists.sourceforge.net/lists/listinfo/modules-interest
>> >>
>> >
>> >
>> >
>> > _______________________________________________
>> > Modules-interest mailing list
>> > Mod...@li...
>> > https://lists.sourceforge.net/lists/listinfo/modules-interest
>> >
>>
>> --
>> --------------------------------------------------------
>> The views and opinions expressed above are strictly
>> those of the author(s).  The content of this message has
>> not been reviewed nor approved by any entity whatsoever.
>> --------------------------------------------------------
>> Paul FM         Info: http://umn.edu/~paulfm/
>> --------------------------------------------------------
>>
>>
>> _______________________________________________
>> Modules-interest mailing list
>> Mod...@li...
>> https://lists.sourceforge.net/lists/listinfo/modules-interest
>>
> _______________________________________________
> Modules-interest mailing list
> Mod...@li...
> https://lists.sourceforge.net/lists/listinfo/modules-interest
>

Re: [Modules] module switch not preserving order

[Roberto B. <Rob...@pi...>]

I agree that the new behaviour is much better.
Nevertheless i understand that if you have many files that rely on the old
behaviour it would be best to have a default option (that can be set by the
user, or the modulefiles maintainer for all users using the modules) that
has compatibility with the "old" way of dealing with this.

Roberto

On Fri, Jun 26, 2020 at 2:54 PM Paul FM via Modules-interest <
mod...@li...> wrote:

> You could add an option to switch/swap to choose this behavior.
>
> -o  (for preserve module order)
> -n  (for NO preserve module order)
> -i  (for do it the old inconsistent way)
>
> Then have something in the config that sets the default.
> (and maybe have module -h list the defaults).
>
> Personally, I like the new behavior (I was not even really aware of the
> old behavior).  The new behavior is more consistent with what I would
> expect.
>
> There are always going to be some modules which don't unload very well and
> if switch were to unload and reload those modules there could be side
> effects (like removing something from the environment that was set by
> something other than a module).
>
>
> --  On 2020-06-26 1:32 AM, Xavier Delaruelle wrote:
> > Hi Leo,
> >
> > What you encounter is the expected behavior starting version 4. I have
> > described this difference of behavior for the switch sub-command on the
> > diff doc [1].
> >
> > The switch in place of version 3 was very specific by replacing
> append-path
> > of prepend-path entries from the switched-on module at the exact same
> place
> > entries are removed for the switched-off module. It coped well when
> > switching modules which are really close but could lead to
> inconsistencies
> > when those 2 modules are different, like for instance if you want to
> switch
> > a compiler by another.
> >
> > One goal of version 4 is to bring consistency over the module actions,
> that
> > is why the behavior of the switch sub-command from version 3 has not been
> > ported to version 4.
> >
> > However it seems possible to have this behavior back if every loaded
> > modules after switched-off module are automatically unloaded, then
> > switched-on module is loaded and after that every automatically unloaded
> > modules are loaded again. This could be opt-in throught an option to have
> > switch behaves as "switch in place".
> >
> > Best regards,
> > Xavier
> >
> > [1]
> >
> https://modules.readthedocs.io/en/latest/diff_v3_v4.html#module-sub-commands
> >
> > Le jeu. 25 juin 2020 à 02:41, Leo Butler <leo...@gm...> a écrit :
> >
> >> Hi Xavier,
> >>
> >> I think perhaps this question has been asked and answered before but
> I've
> >> searched the archive and looked in MIGRATING.txt but haven't found it.
> If
> >> it's a duplicate, my apologies.
> >>
> >> Using the current version of modules (v4.5.1), loaded module order isn't
> >> preserved. To demonstrate, I set up four modulefiles that look like
> this:
> >>
> >> alpha/1: append-path TEST_VAR alpha1
> >> beta/2: append-path TEST_VAR beta2
> >> beta/4: append-path TEST_VAR beta4
> >> gamma/3: append-path TEST_VAR gamma3
> >>
> >> % module load alpha beta/2 gamma
> >> % module list
> >> Currently Loaded Modulefiles:
> >>   1) alpha/1   2) beta/2   3) gamma/3
> >> % echo $TEST_VAR
> >> alpha1:beta2:gamma3
> >> % module switch beta/4
> >> % module list
> >> Currently Loaded Modulefiles:
> >>   1) alpha/1   2) gamma/3   3) beta/4
> >> % echo $TEST_VAR
> >> alpha1:gamma3:beta4
> >>
> >> With v3.2, the original order is preserved, and I think this is the
> >> desired behavior.
> >>
> >> % module list
> >> Currently Loaded Modulefiles:
> >>    1) alpha/1   2) beta/4    3) gamma/3
> >> % echo $TEST_VAR
> >> alpha1:beta4:gamma3
> >>
> >> Anyone looking to move a modulefile into a different place in the order
> >> could simply do a load followed by an unload. The switch command seems
> to
> >> just be a proxy for this in v4.x, at least by default.
> >>
> >> I've looked for something to alter this behavior but I've not found it.
> >> Any suggestions?
> >>
> >> Thanks,
> >>
> >> - Leo
> >> _______________________________________________
> >> Modules-interest mailing list
> >> Mod...@li...
> >> https://lists.sourceforge.net/lists/listinfo/modules-interest
> >>
> >
> >
> >
> > _______________________________________________
> > Modules-interest mailing list
> > Mod...@li...
> > https://lists.sourceforge.net/lists/listinfo/modules-interest
> >
>
> --
> --------------------------------------------------------
> The views and opinions expressed above are strictly
> those of the author(s).  The content of this message has
> not been reviewed nor approved by any entity whatsoever.
> --------------------------------------------------------
> Paul FM         Info: http://umn.edu/~paulfm/
> --------------------------------------------------------
>
>
> _______________________________________________
> Modules-interest mailing list
> Mod...@li...
> https://lists.sourceforge.net/lists/listinfo/modules-interest
>

Re: [Modules] module switch not preserving order

[Paul FM <pa...@um...>]

You could add an option to switch/swap to choose this behavior.

-o  (for preserve module order)
-n  (for NO preserve module order)
-i  (for do it the old inconsistent way)

Then have something in the config that sets the default.
(and maybe have module -h list the defaults).

Personally, I like the new behavior (I was not even really aware of the old behavior).  The new behavior is more consistent with what I would expect.

There are always going to be some modules which don't unload very well and if switch were to unload and reload those modules there could be side effects (like removing something from the environment that was set by something other than a module).


--  On 2020-06-26 1:32 AM, Xavier Delaruelle wrote:
> Hi Leo,
> 
> What you encounter is the expected behavior starting version 4. I have
> described this difference of behavior for the switch sub-command on the
> diff doc [1].
> 
> The switch in place of version 3 was very specific by replacing append-path
> of prepend-path entries from the switched-on module at the exact same place
> entries are removed for the switched-off module. It coped well when
> switching modules which are really close but could lead to inconsistencies
> when those 2 modules are different, like for instance if you want to switch
> a compiler by another.
> 
> One goal of version 4 is to bring consistency over the module actions, that
> is why the behavior of the switch sub-command from version 3 has not been
> ported to version 4.
> 
> However it seems possible to have this behavior back if every loaded
> modules after switched-off module are automatically unloaded, then
> switched-on module is loaded and after that every automatically unloaded
> modules are loaded again. This could be opt-in throught an option to have
> switch behaves as "switch in place".
> 
> Best regards,
> Xavier
> 
> [1]
> https://modules.readthedocs.io/en/latest/diff_v3_v4.html#module-sub-commands
> 
> Le jeu. 25 juin 2020 à 02:41, Leo Butler <leo...@gm...> a écrit :
> 
>> Hi Xavier,
>>
>> I think perhaps this question has been asked and answered before but I've
>> searched the archive and looked in MIGRATING.txt but haven't found it. If
>> it's a duplicate, my apologies.
>>
>> Using the current version of modules (v4.5.1), loaded module order isn't
>> preserved. To demonstrate, I set up four modulefiles that look like this:
>>
>> alpha/1: append-path TEST_VAR alpha1
>> beta/2: append-path TEST_VAR beta2
>> beta/4: append-path TEST_VAR beta4
>> gamma/3: append-path TEST_VAR gamma3
>>
>> % module load alpha beta/2 gamma
>> % module list
>> Currently Loaded Modulefiles:
>>   1) alpha/1   2) beta/2   3) gamma/3
>> % echo $TEST_VAR
>> alpha1:beta2:gamma3
>> % module switch beta/4
>> % module list
>> Currently Loaded Modulefiles:
>>   1) alpha/1   2) gamma/3   3) beta/4
>> % echo $TEST_VAR
>> alpha1:gamma3:beta4
>>
>> With v3.2, the original order is preserved, and I think this is the
>> desired behavior.
>>
>> % module list
>> Currently Loaded Modulefiles:
>>    1) alpha/1   2) beta/4    3) gamma/3
>> % echo $TEST_VAR
>> alpha1:beta4:gamma3
>>
>> Anyone looking to move a modulefile into a different place in the order
>> could simply do a load followed by an unload. The switch command seems to
>> just be a proxy for this in v4.x, at least by default.
>>
>> I've looked for something to alter this behavior but I've not found it.
>> Any suggestions?
>>
>> Thanks,
>>
>> - Leo
>> _______________________________________________
>> Modules-interest mailing list
>> Mod...@li...
>> https://lists.sourceforge.net/lists/listinfo/modules-interest
>>
> 
> 
> 
> _______________________________________________
> Modules-interest mailing list
> Mod...@li...
> https://lists.sourceforge.net/lists/listinfo/modules-interest
> 

-- 
--------------------------------------------------------
The views and opinions expressed above are strictly
those of the author(s).  The content of this message has
not been reviewed nor approved by any entity whatsoever.
--------------------------------------------------------
Paul FM         Info: http://umn.edu/~paulfm/
--------------------------------------------------------

Re: [Modules] module switch not preserving order

[GUENNEUGUES, O. <oli...@at...>]

Hi Xavier,

I would expect the same behavior as described by Leo.
A “switch in place” option would be great!

Thanks for your great  work!

De : Xavier Delaruelle <xav...@gm...>
Envoyé : vendredi 26 juin 2020 08:32
À : Environment Modules usage and discussion. <mod...@li...>
Objet : Re: [Modules] module switch not preserving order


Caution! External email. Do not open attachments or click links, unless this email comes from a known sender and you know the content is safe.
Hi Leo,

What you encounter is the expected behavior starting version 4. I have described this difference of behavior for the switch sub-command on the diff doc [1].

The switch in place of version 3 was very specific by replacing append-path of prepend-path entries from the switched-on module at the exact same place entries are removed for the switched-off module. It coped well when switching modules which are really close but could lead to inconsistencies when those 2 modules are different, like for instance if you want to switch a compiler by another.

One goal of version 4 is to bring consistency over the module actions, that is why the behavior of the switch sub-command from version 3 has not been ported to version 4.

However it seems possible to have this behavior back if every loaded modules after switched-off module are automatically unloaded, then switched-on module is loaded and after that every automatically unloaded modules are loaded again. This could be opt-in throught an option to have switch behaves as "switch in place".

Best regards,
Xavier

[1] https://modules.readthedocs.io/en/latest/diff_v3_v4.html#module-sub-commands<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmodules.readthedocs.io%2Fen%2Flatest%2Fdiff_v3_v4.html%23module-sub-commands&data=02%7C01%7Colivier.guenneugues%40atos.net%7C99e1990731674a28c9f908d8199acfab%7C33440fc6b7c7412cbb730e70b0198d5a%7C0%7C0%7C637287499986410760&sdata=hCYlqYcfw%2F80Y%2BzqhhDhnhAb92uuaWF7I4GpvkSpWP8%3D&reserved=0>

Le jeu. 25 juin 2020 à 02:41, Leo Butler <leo...@gm...<mailto:leo...@gm...>> a écrit :
Hi Xavier,

I think perhaps this question has been asked and answered before but I've searched the archive and looked in MIGRATING.txt but haven't found it. If it's a duplicate, my apologies.

Using the current version of modules (v4.5.1), loaded module order isn't preserved. To demonstrate, I set up four modulefiles that look like this:

alpha/1: append-path TEST_VAR alpha1
beta/2: append-path TEST_VAR beta2
beta/4: append-path TEST_VAR beta4
gamma/3: append-path TEST_VAR gamma3

% module load alpha beta/2 gamma
% module list
Currently Loaded Modulefiles:
 1) alpha/1   2) beta/2   3) gamma/3
% echo $TEST_VAR
alpha1:beta2:gamma3
% module switch beta/4
% module list
Currently Loaded Modulefiles:
 1) alpha/1   2) gamma/3   3) beta/4
% echo $TEST_VAR
alpha1:gamma3:beta4

With v3.2, the original order is preserved, and I think this is the desired behavior.

% module list
Currently Loaded Modulefiles:
  1) alpha/1   2) beta/4    3) gamma/3
% echo $TEST_VAR
alpha1:beta4:gamma3

Anyone looking to move a modulefile into a different place in the order could simply do a load followed by an unload. The switch command seems to just be a proxy for this in v4.x, at least by default.

I've looked for something to alter this behavior but I've not found it. Any suggestions?

Thanks,

- Leo
_______________________________________________
Modules-interest mailing list
Mod...@li...<mailto:Mod...@li...>
https://lists.sourceforge.net/lists/listinfo/modules-interest<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.sourceforge.net%2Flists%2Flistinfo%2Fmodules-interest&data=02%7C01%7Colivier.guenneugues%40atos.net%7C99e1990731674a28c9f908d8199acfab%7C33440fc6b7c7412cbb730e70b0198d5a%7C0%7C0%7C637287499986410760&sdata=gmuHdxyxewg9QdzmJsAEmi0JXW%2FpJCY%2FDM5uQ9XxNEY%3D&reserved=0>

Re: [Modules] module switch not preserving order

[Xavier D. <xav...@gm...>]

Hi Leo,

What you encounter is the expected behavior starting version 4. I have
described this difference of behavior for the switch sub-command on the
diff doc [1].

The switch in place of version 3 was very specific by replacing append-path
of prepend-path entries from the switched-on module at the exact same place
entries are removed for the switched-off module. It coped well when
switching modules which are really close but could lead to inconsistencies
when those 2 modules are different, like for instance if you want to switch
a compiler by another.

One goal of version 4 is to bring consistency over the module actions, that
is why the behavior of the switch sub-command from version 3 has not been
ported to version 4.

However it seems possible to have this behavior back if every loaded
modules after switched-off module are automatically unloaded, then
switched-on module is loaded and after that every automatically unloaded
modules are loaded again. This could be opt-in throught an option to have
switch behaves as "switch in place".

Best regards,
Xavier

[1]
https://modules.readthedocs.io/en/latest/diff_v3_v4.html#module-sub-commands

Le jeu. 25 juin 2020 à 02:41, Leo Butler <leo...@gm...> a écrit :

> Hi Xavier,
>
> I think perhaps this question has been asked and answered before but I've
> searched the archive and looked in MIGRATING.txt but haven't found it. If
> it's a duplicate, my apologies.
>
> Using the current version of modules (v4.5.1), loaded module order isn't
> preserved. To demonstrate, I set up four modulefiles that look like this:
>
> alpha/1: append-path TEST_VAR alpha1
> beta/2: append-path TEST_VAR beta2
> beta/4: append-path TEST_VAR beta4
> gamma/3: append-path TEST_VAR gamma3
>
> % module load alpha beta/2 gamma
> % module list
> Currently Loaded Modulefiles:
>  1) alpha/1   2) beta/2   3) gamma/3
> % echo $TEST_VAR
> alpha1:beta2:gamma3
> % module switch beta/4
> % module list
> Currently Loaded Modulefiles:
>  1) alpha/1   2) gamma/3   3) beta/4
> % echo $TEST_VAR
> alpha1:gamma3:beta4
>
> With v3.2, the original order is preserved, and I think this is the
> desired behavior.
>
> % module list
> Currently Loaded Modulefiles:
>   1) alpha/1   2) beta/4    3) gamma/3
> % echo $TEST_VAR
> alpha1:beta4:gamma3
>
> Anyone looking to move a modulefile into a different place in the order
> could simply do a load followed by an unload. The switch command seems to
> just be a proxy for this in v4.x, at least by default.
>
> I've looked for something to alter this behavior but I've not found it.
> Any suggestions?
>
> Thanks,
>
> - Leo
> _______________________________________________
> Modules-interest mailing list
> Mod...@li...
> https://lists.sourceforge.net/lists/listinfo/modules-interest
>

[Modules] module switch not preserving order

[Leo B. <leo...@gm...>]

Hi Xavier,

I think perhaps this question has been asked and answered before but I've
searched the archive and looked in MIGRATING.txt but haven't found it. If
it's a duplicate, my apologies.

Using the current version of modules (v4.5.1), loaded module order isn't
preserved. To demonstrate, I set up four modulefiles that look like this:

alpha/1: append-path TEST_VAR alpha1
beta/2: append-path TEST_VAR beta2
beta/4: append-path TEST_VAR beta4
gamma/3: append-path TEST_VAR gamma3

% module load alpha beta/2 gamma
% module list
Currently Loaded Modulefiles:
 1) alpha/1   2) beta/2   3) gamma/3
% echo $TEST_VAR
alpha1:beta2:gamma3
% module switch beta/4
% module list
Currently Loaded Modulefiles:
 1) alpha/1   2) gamma/3   3) beta/4
% echo $TEST_VAR
alpha1:gamma3:beta4

With v3.2, the original order is preserved, and I think this is the desired
behavior.

% module list
Currently Loaded Modulefiles:
  1) alpha/1   2) beta/4    3) gamma/3
% echo $TEST_VAR
alpha1:beta4:gamma3

Anyone looking to move a modulefile into a different place in the order
could simply do a load followed by an unload. The switch command seems to
just be a proxy for this in v4.x, at least by default.

I've looked for something to alter this behavior but I've not found it. Any
suggestions?

Thanks,

- Leo

[Modules] Modules project's logo

[<Xav...@CE...>]

Hi All,

This is my great pleasure to present you the new visual identity of Modules.

[cid:159...@u-...]

This logo features an isometric shape representing cubes, organized as the "M" letter of Modules, inside a cube. These internal cubes recall modulefiles aggregated to shape a coherent environment (the surrounding cube). This logo is red to express health, energy and action. I also felt like too much logo are blue nowadays so red is good to stand out :-D. The overall design is smooth and simple to emphasis such properties of the "module" tool.

I would like to thanks my employer, CEA, which has sponsored this design work.

Cheers,
Xavier

[Modules] Modules 4.5.1 released

[Xavier D. <xav...@gm...>]

Hello All,

Version 4.5.1 of Modules has just been released. It provides fixes for the
few issues recently spotted.

The changes introduced in this bugfix release are detailed at:


https://timeline.noticeable.io/cbzeG7wTvAIqj21zbUmx/posts/modules-4-5-1-released

The list of all changes made in this new version are described in the NEWS
document. Details on the new features introduced in the 4.5 series are
provided in the MIGRATING document.

 http://modules.readthedocs.io/en/stable/MIGRATING.html
 http://modules.readthedocs.io/en/stable/NEWS.html

The tarball of this new version can be downloaded at:

 http://downloads.sourceforge.net/modules/modules-4.5.1.tar.gz

The zipball to install this new version on Windows platform can be
downloaded at:

 http://downloads.sourceforge.net/modules/modules-4.5.1-win.zip

If you encounter any issue, please let us know by creating a ticket on the
project bug tracker at:

 https://github.com/cea-hpc/modules/issues

Special thanks to Roy Storey for his code contribution on this release.
Many thanks also to all of you who improve Modules by reporting bugs or
sharing new ideas.

Best regards,
Xavier

Re: [Modules] Reviving the thread: Performance impact from polling module files

[Xavier D. <xav...@gm...>]

Interesting topic. I would suggest you to checkout the git repository of
the project [1] and try your patch on it.

Since v4.1 there have been many improvements on this performance topic,
especially with Modules Tcl extension library [2].

The script/mt utility (make test) will help you to check if the change
introduces regression. Coverage is quite good so if there is an issue with
the change, the testsuite will tell you.

To evaluate how the performances of modulecmd.tcl evolves between releases,
I have made few months ago the script/mb utility (make bench). This script
compares execution times of latest version of Modules v4.1 ... v4.5 and
current git branch over a series of basic commands (avail, list, whatis,
load, ...). It could help you to check how your change impacts performances.

Regards,
Xavier

[1] https://github.com/cea-hpc/modules
[2]
https://modules.readthedocs.io/en/latest/MIGRATING.html#further-i-o-operations-optimization

Le mer. 20 mai 2020 à 20:59, drew stortz <dre...@gm...> a écrit :

> If you don't mind me digging up this thread from 6 years ago... I'm using a modern-ish version of modules (4.1.2) and noticed the same accesses to unused modules versions, and the performance impact is significant. I haven't been able to find the 'other thread' from Rob, but I have found that the following 2 line hack to modulecmd.tcl appears to prevent the extra accesses without any impact to functionality that I've been able to detect so far.
>
> 2679c2679
> <          array set mod_list [getModules $dir $modroot 0 "rc_defs_included"\
> ---
> >          array set mod_list [getModules $dir $mod 0 "rc_defs_included"\
> 4183c4183
> <       array set found_list [findModules $dir $findmod $fetch_mtime\
> ---
> >       array set found_list [findModules $dir $mod $fetch_mtime\
>
> Though I have to admit that I haven't yet tried running the included regressions with this change. If anyone is still interested in this topic, I'd be very interested to discuss it further and compare notes.
>
>
> On Tue, 20 May 2014, Paul FM wrote:
>
> > This is not a direct answer to your question (I am not one of the module
> > maintainers).
> >
> > But rather a suggested work-around.
> >
> > Since modules files take fairly little space - I recommend you use rsync
> > ( http://rsync.samba.org/ ) or something similar (if your OS can't run
> > rsync) to keep copies on the local machines
> [...]
>
> An update; I also received emails off-list from folks saying they were
> using workarounds of copying modulefiles to RAM disks etc..
>
> Instead, my colleague Rob has prepared that I hope substantially reduces
> the need for such workarounds -- if you are interested to try it's in a
> new thread.
>
> The patch fixes the performance 'bug' by removing stat'ing and reading of
> module files that aren't going to be loaded.
>
> With this we found a significant performance increase.
>
> It is noticable when loading (from cold) lots of modules even from a fast
> network drive, and gives a huge improvement where the network drive is of
> higher latency.
>
> Of course different people have other reasons for wanting to localise
> module files (eg. as part of a larger pool of software)
>
> Thanks
>
> --
> Mark
>
>
> _______________________________________________
> Modules-interest mailing list
> Mod...@li...
> https://lists.sourceforge.net/lists/listinfo/modules-interest
>

[Modules] Reviving the thread: Performance impact from polling module files

[drew s. <dre...@gm...>]

If you don't mind me digging up this thread from 6 years ago... I'm
using a modern-ish version of modules (4.1.2) and noticed the same
accesses to unused modules versions, and the performance impact is
significant. I haven't been able to find the 'other thread' from Rob,
but I have found that the following 2 line hack to modulecmd.tcl
appears to prevent the extra accesses without any impact to
functionality that I've been able to detect so far.

2679c2679
<          array set mod_list [getModules $dir $modroot 0 "rc_defs_included"\
---
>          array set mod_list [getModules $dir $mod 0 "rc_defs_included"\
4183c4183
<       array set found_list [findModules $dir $findmod $fetch_mtime\
---
>       array set found_list [findModules $dir $mod $fetch_mtime\

Though I have to admit that I haven't yet tried running the included
regressions with this change. If anyone is still interested in this
topic, I'd be very interested to discuss it further and compare notes.


On Tue, 20 May 2014, Paul FM wrote:

> This is not a direct answer to your question (I am not one of the module
> maintainers).
>
> But rather a suggested work-around.
>
> Since modules files take fairly little space - I recommend you use rsync
> ( http://rsync.samba.org/ ) or something similar (if your OS can't run
> rsync) to keep copies on the local machines
[...]

An update; I also received emails off-list from folks saying they were
using workarounds of copying modulefiles to RAM disks etc..

Instead, my colleague Rob has prepared that I hope substantially reduces
the need for such workarounds -- if you are interested to try it's in a
new thread.

The patch fixes the performance 'bug' by removing stat'ing and reading of
module files that aren't going to be loaded.

With this we found a significant performance increase.

It is noticable when loading (from cold) lots of modules even from a fast
network drive, and gives a huge improvement where the network drive is of
higher latency.

Of course different people have other reasons for wanting to localise
module files (eg. as part of a larger pool of software)

Thanks

-- 
Mark

[Modules] Modules@latest (aka 4.5.0) released

[Xavier D. <xav...@gm...>]

Hello All,

This is it. After 3 months of development, version 4.5.0 of Modules is
available. It is as always backward-compatible with previous version 4
releases and primarily adds features and fixes bugs. Major new features
introduced in v4.5.0 since v4.4 are:

* ml command
* JSON format output
* Improved Windows support
* Error stack trace
* Automatic default and latest symbolic versions

Details on these features included in this new release are provided in the
MIGRATING document. The list of all changes made in this new version are
described in the NEWS document.

 http://modules.readthedocs.io/en/stable/MIGRATING.html
 http://modules.readthedocs.io/en/stable/NEWS.html

The tarball of this new version can be downloaded at:

 http://downloads.sourceforge.net/modules/modules-4.5.0.tar.gz

*NEW* The zipball to install this new version on Windows platform can be
downloaded at:

 http://downloads.sourceforge.net/modules/modules-4.5.0-win.zip

If you encounter any issue, please let us know by creating a ticket on the
project bug tracker at:

 https://github.com/cea-hpc/modules/issues

Special thanks to Colin Marquardt, Jacques Raphanel, Andrey Portnoy, Armin
Wehrfritz, Tom Payerle, Adrien Cotte, Dominique Martinet and Martin
Schuster for their code contribution on this release. Many thanks also to
all of you who improve Modules by reporting bugs or sharing new ideas.

Best regards,
Xavier

[Modules] Automatic default and latest symbolic versions

[Xavier D. <xav...@gm...>]

When the implicit default mechanism and the *advanced module version
specifiers* [1] are both enabled, a `default` and a `latest` symbolic
versions are automatically defined for each module name.

This new feature gives the ability to select the highest version available
for a module, without knowing beforehand this version name:

    $ module load -v foo@latest
    Loading foo/1.10

The symbolic versions are automatically defined unless a symbolic version,
an alias or a regular module version already exists for these `default` or
`latest` version names.

This new feature will be part of the next version (v4.5), whose release is
imminent.

Regards,
Xavier

[1]
https://modules.readthedocs.io/en/latest/MIGRATING.html#advanced-module-version-specifiers

[Modules] Error stack trace

[Xavier D. <xav...@gm...>]

Error messages will now embed a stack trace for unknown errors to help
localize the root cause of issues. This change applies to modulefile
evaluation:

    Loading foo/1.2
      Module ERROR: add-path cannot handle path equals to separator string
            while executing
        "append-path PATH :"
            (file "/path/to/modulefiles/foo/1.2" line 24)
        Please contact <root@localhost>

A stack trace is also returned when an unknown error occurs in
`modulecmd.tcl` script, which facilitates issue report and analysis:

    $ module load bar
    ERROR: invalid command name "badcommand"
          while executing
      "badcommand"
          (procedure "module" line 14)
          invoked from within
      "module load bar"
          ("eval" body line 1)
          invoked from within
      "eval $execcmdlist"
      Please report this issue at https://github.com/cea-hpc/modules/issues

This new feature is available in git repository and will be included into
the upcoming v4.5 release.

Regards,
Xavier

Re: [Modules] LD_PRELOAD and MODULES_RUN_QUARANTINE results in warning messages

[Xavier D. <xav...@gm...>]

Hi James,

I have applied the proposed change to the RPM spec file proposed by the
project [1]. I will advertise this change when the 4.5 version of Modules
will be released toward Fedora package maintainer.

If you got this issue on a specific Linux distribution like EL8 or
Debian/Ubuntu, please report this issue to their bug tracker. For them to
see there is a need to update the configuration in their packaging of
Modules.

Best regards,
Xavier

[1]
https://github.com/cea-hpc/modules/commit/3c6932365f57f4adbfae4dd46c4efa8f1c7e9046

Le ven. 27 mars 2020 à 18:09, Xavier Delaruelle <xav...@gm...>
a écrit :

> Hi James,
>
> I have initially skipped LD_PRELOAD from the default quarantine variable
> list as we use it locally to add libs that track system usages.
>
> But seeing the issue you get it seems more coherent to also have
> LD_PRELOAD part of the list along with LD_LIBRARY_PATH.
>
> I need a bit more time to think of it. But my first thought is that such
> change should be harmless. And if people need to keep LD_PRELOAD value even
> for tclsh, they will still be able to change that locally.
>
> Best regards,
> Xavier
>
>
>
> Le ven. 27 mars 2020 à 14:58, James Clark <jam...@ar...> a écrit :
>
>> Hi,
>>
>> We have a tool that uses LD_LIBRARY_PATH and LD_PRELOAD, and we're seeing
>> warnings from the loader when
>> environment module runs tclsh with LD_LIBRARY_PATH cleared due to
>> MODULES_RUN_QUARANTINE.
>>
>> For example:
>>
>>   $ export LD_PRELOAD=somelib.so
>>   $ export LD_LIBRARY_PATH=path/to/somelib/
>>   $ bash
>>
>> Results in this error message:
>>   ERROR: ld.so: object 'somelib.so' from LD_PRELOAD cannot be preloaded
>> (cannot open shared object file): ignored.
>>
>>
>> Were going to add a workaround in our tool that appends LD_PRELOAD to
>> MODULES_RUN_QUARANTINE. But the default
>> setup for bash only has LD_LIBRARY_PATH cleared. Is it possible to update
>> the defaults to also clear LD_PRELOAD as well?
>>
>>
>> Thanks
>> James
>>
>>
>> _______________________________________________
>> Modules-interest mailing list
>> Mod...@li...
>> https://lists.sourceforge.net/lists/listinfo/modules-interest
>>
>