|
From: Robert M. <mc...@ta...> - 2008-10-16 20:39:23
|
Summary: ------------- I have created yet another module system (Lmod) to handle what I call the "MODULEPATH hierarchy problem". We have a number of compilers and libraries. Unfortunately libraries compiled with one compiler are not always compatible with code compiled by another. So users will chose a compiler and one or more libraries defined by modules. The problem comes in when a user wants to change compilers. With the current module system, they have to unload the compiler module plus any compiler dependent modules. Then they have to load the new compiler module plus any library modules they need. The Lmod module system unloads or reloads modules as the MODULEPATH changes automatically. I am sending this announcement to the modules.org list to see if others have this problem and would be interested. I am looking for a place to host "Lmod" and "lua" with all the required libraries. Lmod has the same MIT licence that lua has. I'll be happy to answer any questions. Announcement: ------------ I am announcing a new lua-based module system named "Lmod". The first question anyone reading this should ask is "Why would anyone come up with yet another version of modules, written in a language that not many people have heard of?" Well, there are many reasons why but the first is that I'm not fond of TCL and the problem I'm trying to solve would probably require a complete rewrite of the TCL module system. Starting from scratch was much simpler than trying to figure out either of the three current versions of modules (Modules 3.+, TCL Modules, C-Modules). The Problem: ------------ Here at TACC (the Texas Advanced Computing Center) we have large parallel computer systems. On our newest system, we support several compiler families (pgi, intel, ...) and different MPI Libraries (Mvapich 1 and 2, Openmpi, ...) and libraries that depend on the compiler and Mpi library such as PETSc. So a user application which links against a high-level library such as PETSc implicitly depends on a particular compiler and MPI library in addition to PETSc. To handle this, the compiler module adds to the MODULEPATH which points to the possible MPI libraries. Each MPI module then adds to the MODULEPATH for modules that are available for that compiler and mpi choice. To change compilers, a user must unload PETSc, the mpi library and the compiler, and then load the new compiler, new mpi library, etc. Clearly the current version of the module command makes this more complicated than it needs to be. My Solution: ------------ The module command knows all the modules which are currently loaded and where the module file is located. My new module command watches for changes to the environment variable MODULEPATH. If MODULEPATH changes, then all modules are checked to see if module and module file still match. Any that do not have the old module unloaded and the same-named module in the new location loaded. If a user loads a compiler and mpi library, then unloads the compiler, the mpi library module will not be found. In this case, the mpi module is marked as inactive. Any subsequent module commands that load a compiler or any module that changes the MODULEPATH will force Lmod to try and reload inactive modules. There are of course complications about detecting which modules have been automatically inactivated versus those which a user has explicitly requested to remove. In the case where a user swaps compiler modules, any mpi modules that have been reloaded are reported to the user. Why Lua? -------- Lua is a language that was originally designed to fill the same niche as TCL. Namely, it is designed to be used as a library that running programs can use as a configuration language. It is from the Pontifical Catholic University of Rio de Janeiro in Brazil. The name Lua is Portuguese for Moon. Lua is a small, fast, portable scripting language. Its main data structure are hash tables which are extended to support many things including Object-Oriented Programming. It is similar to Python but is much smaller and easier to both embed and extend. It is now used in many application areas. It is quite popular in the computer gaming industry. For example, World of Warcraft uses lua to drive the game engine. See http://www.lua.org for more information. One of the useful features of Lua is that functions are first class objects. That means that functions can be handled like variables: they can be copied and renamed. This is a huge advantage when dealing with the the "load" function inside a module. When loading a module, the "load" function equals the "Load" function which evaluates and sets environment variables. When unloading, the "load" function equals the "UnLoad" function which does the reverse. All the other functions, such as setenv and prepend_path, are handled similarly. This simplifies the design as each the functions only has to handle one operation. TCL Modules ----------- Obviously, Lmod natively expects lua encoded modulefile. However, since rewriting all the modulefiles we have is a big task, Lmod supports reading TCL based modules. If a module has a .lua extension then it is evaluated by Lmod directly. All others are assumed to be written in TCL. A TCL module is evaluated by a small TCL program which prints out lua command replacements for the TCL ones. This output is captured by Lmod and evaluated. This performs quite well, and therefore adoption of the Lmod system is not hindered by the requirement to replace a large installed base of TCL modules with lua ones. Current Differences between Lmod and Modules 3+: ----------------------------------------------- The output of Lmod is compatible with use under Zsh, Bash, Tcsh, and Csh. It currently doesn't support all the other shell output choices but these could be added if there is interest. |
