File	Date	Author	Commit
LICENSE	2013-04-16	Mark Dixon	[0bc1b5] Initial commit
README	2022-10-20	Mark Dixon	[f74af4] Updated copyright statement in README
flavours.tcl	2022-10-20	Mark Dixon	[5f56e3] flavours.tcl - commented-out a diagnostic line
modulecmd.wrapper	2013-04-16	Mark Dixon	[0bc1b5] Initial commit
pkgIndex.tcl	2013-04-16	Mark Dixon	[0bc1b5] Initial commit

Read Me

"flavours" version 1.0.0
========================

License:
=======

Copyright (C) 2008-2013 University of Leeds
Copyright (C) 2022 University of Durham

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program.  If not, see <http://www.gnu.org/licenses/>.


The problem:
===========

Before writing this extension, we used to have lots and lots of modules
with similar names. An example of such a scheme might be:

  $ module avail

  ------------------------ /apps/Modules -------------------------
  gnu/4.7.1
  intel/12.1.5.339
  openmpi/1.6.1/32bit/gnu4.7.1
  openmpi/1.6.1/64bit/gnu4.7.1
  openmpi/1.6.1/32bit/intel12.1.5.339
  openmpi/1.6.1/64bit/intel12.1.5.339
  hdf5/1.8.9/32bit/gnu4.7.1
  hdf5/1.8.9/64bit/gnu4.7.1
  hdf5/1.8.9/32bit/gnu4.7.1/openmpi/1.6.1
  hdf5/1.8.9/64bit/gnu4.7.1/openmpi/1.6.1
  hdf5/1.8.9/32bit/intel12.1.5.339
  hdf5/1.8.9/64bit/intel12.1.5.339
  hdf5/1.8.9/32bit/intel12.1.5.339/openmpi/1.6.1
  hdf5/1.8.9/64bit/intel12.1.5.339/openmpi/1.6.1

Unsurprisingly, we had frequent complaints from horribly confused users.
"module avail" just scared them, so they never quite knew what was
installed. Over time, as we installed newer compilers and MPI libraries,
this just got worse.

To get round the problem, we tried various tactics - automatic loading of
modules, modules that were collections of other modules, multi-level
module directories, aliases, etc. Again, they all just made things worse.
  
We firmly believed that we need to present all these alternatives, but in
a simpler way, to encourage people to at least try different compilers and
MPI implementations - checking both portability and performance.
  
But we wanted to keep modules.


The solution:
============

flavours resolves this by allowing interdependency between modules and
hiding some of the detail, so the above example might collapse to:

  $ module avail

  ------------ /apps/Modules/developers/architecture -------------  
  bit/32         bit/64
  
  -------------- /apps/Modules/developers/compilers --------------
  gnu/4.7.1      intel/12.1.5.339

  -------------- /apps/Modules/developers/libraries --------------
  openmpi/1.6.1  hdf5/1.8.9   

  
Ideally this capability would be built into the modulecmd program itself.
However, in order to quickly provide a proof-of-concept, we instead wrote
a small Tcl library to be called from modulefiles and a small Perl wrapper
around the vanilla modulecmd C binary.

At its core are alternative, suped-up, "prereq" and "conflict" commands,
differing from the traditional versions in the following areas:

* The concept of a class that modules can belong to if they provide that
functionality (e.g. compiler, mpi).

* The concept of optional prerequisites.
  
* Using "prereq" to automatically generate a "flavour" string, used to
find the correct build in the filesystem.
  
* Storing "prereq" information in an environment variable, allowing
modules to be automatically reloaded if one of their prerequisites are
switched or unloaded.

The problem for the sysadmin is then reduced to installing (or symlinking)
compatible builds to the flavour relevant names. We automated the chugging
through all the permutations (not presented here).

Example of its use:

  $ module purge
  $ module load hdf5
  hdf5/1.8.9 depends on one of the module(s) 'bit'
  $ module load bit/64
  $ module load hdf5
  hdf5/1.8.9 depends on one of the module(s) 'gnu intel'
  $ module load intel
  $ module load hdf5

  $ module purge
  $ module load bit/64 gnu/4.7.1 hdf5/1.8.9
  $ echo $HDF5_HOME
  /apps/developers/libraries/hdf5/1.8.9/1/bit-64-gnu-4.7.1
  $ module switch gnu intel
  $ echo $HDF5_HOME
  /apps/developers/libraries/hdf5/1.8.9/1/bit-64-intel-12.1.5.339
  $ module load openmpi
  $ echo $HDF5_HOME
  /apps/developers/libraries/hdf5/1.8.9/1/bit-64-intel-12.1.5.339-openmpi-1.6.1

Example modulefile for a piece of software called hdf5:

  #%Module 1.0
  
  # Initialise "flavours"
  package require flavours
  flavours init 
  
  proc ModulesHelp { } {
      puts stderr "File format for storing and managing data
  
  Information:
    Homepage : http://www.hdfgroup.org/HDF5/
   "
  }

  module-whatis "File format for storing and managing data "
  
  # Construct flavour name
  flavours prereq bit
  flavours prereq -class compiler
  flavours prereq -optional -class mpi
  flavours conflict hdf5 
  
  # Declare where the flavours are held
  flavours root     /apps/developers/libraries/hdf5/1.8.9
  flavours revision 1   
  flavours commit
  
  # Set environment variables
  setenv HDF5_HOME [flavours path]

  # Prepend to environment variables (paths relative to
  # the directory containing the flavour)
  flavours prepend-path PATH            bin
  flavours prepend-path LIBRARY_PATH    lib
  flavours prepend-path LD_LIBRARY_PATH lib
  flavours prepend-path CPATH           include
  
  # Reload any modules with this as a prerequisite
  flavours cleanup

This would search directory /apps/developers/libraries/hdf5/1.8.9/1 for
a directory named after the flavours impled by the "flavours prereq"
lines and the modules currently loaded.

e.g. subdirectory bit-64-gnu-4.7.1 if modules bit/64 and gnu/4.7.1
were loaded.


Install:
=======

1) Install the C version of "modules" as normal.

2) Install modulecmd wrapper

  * Rename the installed module binary from modulecmd to modulecmd.wrapped

  * Copy the supplied modulecmd.wrapper to the same directory, make sure
  it's executable.

  * Edit modulecmd.wrapper so that the modulecmd variable near the top
  of the file contains the full path to the modulecmd.wrapped binary.

  * Create a symlink in the same directory:
  ln -s modulecmd.wrapper modulecmd

3) Install flavours Tcl library

  * Create a directory to hold the Tcl library

  * Copy the supplied flavours.tcl and pkgIndex.tcl to that directory

  * Review the classes defined near the top of flavours.tcl and edit to
  contain space-delimited lists of the modules belonging to each class
  (omitting any module version numbers) at your site.

  * Add the directory to users' TCLLIBPATH environment variable.

This has been tested on RHEL5/RHEL6 and modules 3.2.9.


Todo:
====

Catch-up with modules functionality added since "flavours" was initially
written:

* Move module class definitions from flavours.tcl to modulecmd.wrapper

* Teach modulecmd.wrapper how to do a one-argument "module switch"
between classes.  e.g. "module switch gnu" would switch between any
compiler module and the default "gnu" module.