|
From: Xavier D. <xav...@gm...> - 2025-11-01 17:22:14
|
Dear Achim, Many thanks for this insighful document. I have made an initial analysis and below are my first thoughts. # Module configuration files As you point there is a strong preference in Modules to have the least possible files to read and even to test for existence. All of this for the sake of IO performance. I also recommend to have only one .modulerc file per modulepath. So I would prefer the version-specific configuration information to be put in .modulerc files. We may analyze what makes this approach more complex for automated config changes and work on this to ease automation. .modulerc file may also contain code that reads a central registry file holding all the configuration for all versions, then renders modulerc commands based on the retrieved configurations. # Life cycle In addition to hide and forbid, it is also possible to tag modulefiles. To keep things generic, "unstable" and "deprecated" may be tags applied to modulefiles. What is missing is a way to say that such tag is hidden and the ability to unveil them but not all the hidden modules. A new config option "avail_hide_tagged" may be added to list all tag that should be hidden on a "module avail". This config option value may be changed by user if they want to see the modules holding these specific tags. # Availability based on hostname or OS Here I would like to challenge if the following existing approaches can already cover these needs: * having specific-modulepaths for the system that only contains the modulefiles compatible with it * using "module-virtual" to spawn the modulefile in the modulepath only if the current system is compatible with it # Path manipulations I am totally convinced by this. Having a new modulefile command like "module-autoenv prefix append|prepend" that if defined tests dirs existence and sets env variables following a mapping defined in a configuration option. Could be a great addition. # Module info messages It is stated that "A version-independent message would be useful" on Modules. module-warn may already target the generic module name to apply message to any version of this module. So if I have understood well the need, it is already fulfilled. # Best, Xavier Le jeu. 16 oct. 2025 à 15:59, Gsell, Achim <ach...@ps...> a écrit : > > Dear Xavier > > Since our GitLab server will be shut down at the end of November, I have moved Pmodules to GitHub. The new URL is > https://github.com/Pmodules/Pmodules > > Documentation is available at: > https://pmodules.github.io/ > > I wrote a short document summarising the required features we would need in Modules to replace Pmodules with Modules+site specific procedures/configuration: > https://github.com/Pmodules/Pmodules/wiki/Differences-between-Pmodules-and-Modules > > Comments/suggestions how we can proceed are very welcome! > > Best regards, > > Achim > > On 13 Sep 2025, at 15:16, Xavier Delaruelle <xav...@gm...> wrote: > > Dear Achim, > > Thanks for these informations. I am following the development of > Pmodules since several years through the PSI gitlab. > > That's great if you would like to contribute here. Welcome :-) > > Best regards, > Xavier > > Le ven. 12 sept. 2025 à 11:39, Gsell, Achim <ach...@ps...> a écrit : > > > Dear Xavier, > > thanks a lot for the answer and solution! > > A little background on why I'm experimenting with this. > > At the Paul Scherrer Institute (PSI), we use Pmodules. Pmodules is essentially a wrapper for Environment Modules 3.2.10 written in Bash. This wrapper implements additional features not available in the old implementation of the Environment Modules. Some of them are now available in the current version, or there is similar functionality available. I started developing Pmodules at a time when the development of the Environment Modules was not progressing. This changed and some of the features in Pmodules - like hierarchical modules - are now available in the Environment Modules. Other features like overlays, Lmod support, availability of modules depending on hostname and/or OS/version are not available. > > Since developing and supporting a special version that is only used at PSI is lot of work, we are considering whether some of the Pmodule for PSI essential extensions could be incorporated into the Tcl Environment Modules. I would like to contribute to the project and have a significant part of my working time available for the implementation of these features. > > Best regards > > Achim > > > > On 12 Sep 2025, at 07:39, Xavier Delaruelle <xav...@gm...> wrote: > > Dear Achim, > > First of all, I made a mistake in my analysis yesterday: defining > module-addpath procedure within the modulefile works just fine. > > The issue you face comes from the Tcl evaluation context. When defined > in the siteconfig.tcl, the module-addpath procedure is evaluated in > the main execution context, not the one specific to modulefile > evaluation. Thus, modulefile commands like prepend-path called from > the main context are not the correct one adapted to the current > evaluation mode ("edit-path-sc" for scan mode or "prepend-path-un" for > unload mode). > > To get the expected behavior, the prepend-path command should be run > from current modulefile evaluation context. It could be achieved by > changing the module-addpath procedure the following way: > > proc module-addpath {path} { > # get name of current module Tcl interp > set itrp [getCurrentModfileInterpName] > > interp eval $itrp prepend-path MODULEPATH $path > } > > I need to update the documentation to explain that along the > siteconfig.tcl modulefile_extra_cmds documentation. > > Best regards, > Xavier > > Le jeu. 11 sept. 2025 à 06:14, Xavier Delaruelle > <xav...@gm...> a écrit : > > > Dear Achim, > > Thanks for your message. I can reproduce the issue you describe on my > side. Even when defining the module-addpath procedure within the > modulefile. > > From a first quick analysis, it does not feel that this is the > indented behavior. I will need to investigate further down to find > what is going on. > > Best regards, > Xavier > > Le mer. 10 sept. 2025 à 10:57, Gsell, Achim <ach...@ps...> a écrit : > > > Dear All, > > I'm currently playing with a hierarchical module structure, which is now supported in version 5.6. If I add a new directory to MODULEPATH in a modulefile itself, everything works fine. But as soon as I move the statement > > prepend-path MODULEPATH $something > > to a procedure inside my siteconfig.tcl file I run into two issues. > > First: The path $something is not removed from MODULEPATH while unloading the module. > > Second: module spider —indepth doesn’t show the modules in $something. > > Example: > > I have the following (simplified) modulefile for gcc/15.2.0: > > $ cat /afs/psi.ch/group/HPCE/Modules/Programming/modulefiles/gcc/15.2.0 > #%Module > > module-help "GCC compiler collection" > module-addpath {/afs/psi.ch/group/HPCE/Modules/Compiler/modulefiles/gcc/15.2.0} > > In my siteconfig.tcl the procedure module-addpath is defined as: > > proc module-addpath { path } { > prepend-path MODULEPATH $path > } > > I have the following modulefiles: > > [gsell@merlin-c-002 ~]$ find /afs/psi.ch/group/HPCE/Modules/*/modulefiles -type f > /afs/psi.ch/group/HPCE/Modules/Compiler/modulefiles/gcc/15.2.0/OpenBLAS/0.3.30 > /afs/psi.ch/group/HPCE/Modules/Programming/modulefiles/gcc/15.2.0 > > and MODULEPATH is set to: > > [gsell@merlin-c-002 ~]$ echo $MODULEPATH > /afs/psi.ch/group/HPCE/Modules/Programming/modulefiles > > The spider command should show the OpenBLAS module, but it doesn’t: > > [gsell@merlin-c-002 ~]$ module spider --indepth > ------------ /afs/psi.ch/group/HPCE/Modules/Programming/modulefiles ------------- > gcc/15.2.0 > > Key: > modulepath > > After loading the gcc module, the OpenBLAS module is available, this is correct: > > [gsell@merlin-c-002 ~]$ module load gcc/15.2.0 > [gsell@merlin-c-002 ~]$ module avail > -------- /afs/psi.ch/group/HPCE/Modules/Compiler/modulefiles/gcc/15.2.0 --------- > OpenBLAS/0.3.30 > > ------------ /afs/psi.ch/group/HPCE/Modules/Programming/modulefiles ------------- > gcc/15.2.0 > Key: > loaded modulepath > But after unloading the gcc module, the OpenBLAS module is still available: > > [gsell@merlin-c-002 ~]$ module unload gcc/15.2.0 > [gsell@merlin-c-002 ~]$ module avail > -------- /afs/psi.ch/group/HPCE/Modules/Compiler/modulefiles/gcc/15.2.0 --------- > OpenBLAS/0.3.30 > > ------------ /afs/psi.ch/group/HPCE/Modules/Programming/modulefiles ------------- > gcc/15.2.0 > > Key: > modulepath > and the added directory is still in MODULEPATH: > > [gsell@merlin-c-002 ~]$ echo $MODULEPATH > /afs/psi.ch/group/HPCE/Modules/Compiler/modulefiles/gcc/15.2.0:/afs/psi.ch/group/HPCE/Modules/Programming/modulefiles > > > If I replace in the gcc/15.2.0 modulefile > > module-addpath {/afs/psi.ch/group/HPCE/Modules/Compiler/modulefiles/gcc/15.2.0} > > with > > prepend-path MODULEPATH {/afs/psi.ch/group/HPCE/Modules/Compiler/modulefiles/gcc/15.2.0} > > everything works as expected. > > I’m not sure whether this behaviour is intended or a bug. If it is intended, how can I add something to MODULEPATH in a procedure? > > Best regards, > > Achim > > > > _______________________________________________ > Modules-interest mailing list > Mod...@li... > https://lists.sourceforge.net/lists/listinfo/modules-interest > > > > _______________________________________________ > Modules-interest mailing list > Mod...@li... > https://lists.sourceforge.net/lists/listinfo/modules-interest > > > > _______________________________________________ > Modules-interest mailing list > Mod...@li... > https://lists.sourceforge.net/lists/listinfo/modules-interest > > > > _______________________________________________ > Modules-interest mailing list > Mod...@li... > https://lists.sourceforge.net/lists/listinfo/modules-interest > > > _______________________________________________ > Modules-interest mailing list > Mod...@li... > https://lists.sourceforge.net/lists/listinfo/modules-interest |
