Re: [Modules] Get list of available (virtual) modules from within .modulerc

[Bloecker, M. <Mar...@am...>]

Hello Xavier,

Thanks for your prompt reply. In my experiments I did miss the redirection of STDERR and I don’t remember if and where I did read about using ‘system’ rather than ‘exec’.  With your tips I was able to come up with a working demonstrator. As discussed some additional measures where necessary to ensure the specific .modulerc file was only processed once. I realized this by temporarily setting an environment variable (from within the .modulerc file) to flag that an instance of it was already being processed. As a reference for whoever might be interested the respective .modulerc looks something like:

#%Module4.7

# get generalized module specific environment variable name
# replacing any directory separators by '_' in the process
set processing_flag_var [regsub -all {/} "MODULERC_PROCESSING_[file dirname [module-info name]]" {_} ]

# evaluate module specific environment variable
# to stop processing if called recursively
if { [info exists ::env($processing_flag_var)] && $::env($processing_flag_var)  } {
    set ::env($processing_flag_var) 0
    next
} else {
    set ::env($processing_flag_var) 1
}

# get list of installed SW modules
set sw_modules [exec 2>@stdout $env(MODULES_CMD) sh avail --terse --output= vendor/software]

foreach sw_version $sw_modules {
    # reduce to version number only
    set sw_version [file tail $sw_version]

    # create new virtual module
    module-virtual "/${sw_version}" .localized-software.module
}

Looking at the complexity of this demonstrator and the associated risks I agreed with your assessment that such an approach is not to be recommended. I will likely go with your proposal to manage both the ‘vendor/software/version’ and ‘localized-software/version’ modules in the same .modulerc file.

To give some further background:
The reason for my initial question was the fact that both modules and all related files are currently managed in two separate git repositories. I wanted to remove the need for synchronized updates of both git repositories whenever a new SW version is installed, namely when the logical content of the modules (apart from changing installation directories) remains the same.

Best Regards,

Martin

From: Xavier Delaruelle <xav...@gm...>
Sent: Mittwoch, 30. März 2022 20:17
To: Environment Modules usage and discussion. <mod...@li...>
Subject: Re: [Modules] Get list of available (virtual) modules from within .modulerc

[---EXTERNAL---]


Le mer. 30 mars 2022 à 14:35, Bloecker, Martin <Mar...@am...<mailto:Mar...@am...>> a écrit :
All,

First of all a big thank you to Xavier and everybody on this list. It is good to see the community efforts to keep environment modules an active project.

Hopefully the community can help me to tackle a problem I’m trying to solve:
I do have a set of dynamically generated modules named (for the sake of abstraction) ‘vendor/software/version’. These are virtual modules that are defined (in vendor/software/.modulerc) by scanning (multiple) SW root directories for the presence of appropriate version-specific subdirectories. This approach works well and a list of created virtual modules can be output via:
module avail --terse --output= software/vendor
Note that these modules will only set the bare minimum environment required for the software. Also not all SW versions might be available on all platforms.

Now for the actual problem:
We need a second set of virtual modules named ‘localized-software/version’ (note the different name space) with a one to one version mapping to the ‘vendor/software’ modules.
As the ‘vendor/software/version’ are generated dynamically per platform the same applies to the ‘localized-software/version’ modules. In order to decouple the ‘vendor/software’ module implementation from the ‘localized-software’ module implementation as much as possible the idea is to use a (filtered) list of available modules (i.e. the list of ‘vendor/software/version’ modules) to determine which virtual ‘localized-software/version’ modules need to be generated.

Is there a way to get a list of (currently) available modules from within a .modulerc file? Unfortunately the intuitive
                set sw_versions [module avail --terse --output= software/vendor]
or
                set sw_versions [system module avail --terse --output= software/vendor]
do not work. Note that the ‘is-avail’ function would not be useful here as one would need to know beforehand which potential versions will exist.

Is there a way to implement what I’m trying to achieve? And a bonus question: In case an implementation is possible, would we need to take precautions against recursively processing the same .modulerc file over and over again? How?

Thanks in advance,

Martin

Hello Martin,

Such code could be written (using MODULES_CMD to locate modulecmd.tcl script):

  set sw_versions [exec 2>@stdout $env(MODULES_CMD) sh avail --terse --output= software/vendor]

But I would not recommend to have such external call in modulerc files, as such processing could be expensive and as you mention there is a risk of infinite processing loop.

I do not totally get the link between software/vendor and localized-software modules, but I would suggest to extend the way the versions of software/vendor are queried to also generate the versions for localized-software modules. Maybe using the .modulerc file at the root of the modulepath could help to generate all virtual modules in the same script, using results of software/vendor creating to also help creating localized-software modules.

Regards,
Xavier