Re: [Lmod-users] Creating an accelerator overlay for a compiler/mpi hierarchy

[Alan O'C. <a....@fz...>]

Looking at  https://lmod.readthedocs.io/en/latest/050_lua_modulefiles.html it would seem that `unload()` will unload regardless of `mode()`. To me that implies that there is no way to trigger a module load when unloading a module since the action of `load()` gets reversed when unloading the module...is there another approach?

On Wed, 26 Aug 2020 at 10:06, Alan O'Cais <a....@fz...<mailto:a....@fz...>> wrote:
Hi all,

I'm trying to see if it is feasible to create a `CUDA` module that creates a GPU-enabled overlay for the normal compiler/mpi hierarchy. My current approach is to (ab)use the Lmod family concept to conditionally expand the hierarchy:
```
local overlay = "/home/alanc/test_cuda/cuda-11/gcccore-10"
prepend_path("MODULEPATH", pathJoin(overlay, "core"))

family("accelerator")

if (os.getenv("LMOD_FAMILY_COMPILER") ~= nil) then
  prepend_path("MODULEPATH", pathJoin(overlay, "compiler", os.getenv("LMOD_FAMILY_COMPILER") .. '-' .. os.getenv("LMOD_FAMILY_COMPILER_VERSION")))
  if (os.getenv("LMOD_FAMILY_MPI") ~= nil) then
    prepend_path("MODULEPATH", pathJoin(overlay, "mpi", os.getenv("LMOD_FAMILY_MPI") .. '-' .. os.getenv("LMOD_FAMILY_MPI_VERSION" .. '-' .. os.getenv("LMOD_FAMILY_COMPILER") .. '-' .. os.getenv("LMOD_FAMILY_COMPILER_VERSION"))
  end
end
```
The above works fine as long as the `CUDA` module is loaded last (with respect to the required level of the hierarchy) and unloaded first, but the other cases is what I was hoping for advice on since it looks like I will need boilerplate in compiler/mpi modules to get the right thing to happen.

I imagine I should be able to trigger a reload of CUDA (using `LMOD_FAMILY_ACCELERATOR(_VERSION)`) when extending the `MODULEPATH` up the hierarchy which would solve issues with loading but I'm a bit lost on how to get the right thing to happen when unloading an element of the hierarchy while CUDA is loaded. I would like to also trigger a reload of CUDA in this case but I need the unload to happen before `LMOD_FAMILY_COMPILER` is unset and the reload to happen afterwards. I was thinking of using
```
prepend_path("MODULEPATH", pathJoin("path/to/compiler/modulepath/extension"))

local accelerator_name = nil
if (os.getenv("LMOD_FAMILY_ACCELERATOR") ~= nil) then
  accelerator_name = os.getenv("LMOD_FAMILY_ACCELERATOR")
  local accelerator_version = os.getenv("LMOD_FAMILY_ACCELERATOR_VERISON")
  if (mode() == "load") then
    unload(pathJoin(accelerator_name, accelerator_version))
  end
  if (mode() == "unload") then
    -- assuming the behaviour of the function is reversed in unload mode
    load(pathJoin(accelerator_name, accelerator_version))
  end
end

family("compiler")

if (accelerator_name ~= nil) then
  if (mode() == "load") then
    load(pathJoin(accelerator_name, accelerator_version))
  end
  if (mode() == "unload") then
    -- assuming the behaviour of the function is reversed in unload mode
    unload(pathJoin(accelerator_name, accelerator_version))
  end
end
```
This code is still a bit rough (and untested) but hopefully the concept is clear, are there any obvious issues with this approach?

One issue I would foresee is that spider may not be able to cache these conditional `MODULEPATH` entries that come from the `CUDA` module.

Another is to understand what happens when the compiler module is unloaded and the mpi module becomes inactive, is the MPI module still unloaded when it is placed in an inactive state? I guess this boils down to asking are there any other responses to `mode()` that I should be accounting for.

Any advice is much appreciated, thanks,

Alan

--
Dr. Alan O'Cais
E-CAM Software Manager
Juelich Supercomputing Centre
Forschungszentrum Juelich GmbH
52425 Juelich, Germany

Phone: +49 2461 61 5213
Fax: +49 2461 61 6656
E-mail: a....@fz...<mailto:a....@fz...>
WWW:    http://www.fz-juelich.de/ias/jsc/EN


--
Dr. Alan O'Cais
E-CAM Software Manager
Juelich Supercomputing Centre
Forschungszentrum Juelich GmbH
52425 Juelich, Germany

Phone: +49 2461 61 5213
Fax: +49 2461 61 6656
E-mail: a....@fz...<mailto:a....@fz...>
WWW:    http://www.fz-juelich.de/ias/jsc/EN


------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Volker Rieke
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------