Environment Modules

> From: "J. Pelan" <J.Pelan@...>
> 
> I was recently made aware of the Modules package when Cray 
> adopted it for use with UNICOS and their mutifarious versions
> of compilers etc.

Cool.  I'm glad to see a vendor using it :-)

> I appears to be an extremely useful tool and one which I'm sure
> many people have thought about developing from time to time.
> Seeing as it is so good why aren't more people aware of it ?

Well, the main reason is that it fixes a problem which is mostly
peculiar to large sites: how to handle user environments with
lots of applications.  Most folks are stuck in either the PC
space (few applications, no concept of multiple users) or the
mainframe space (few applications, many users.)  They have yet
to tackle environments where you have heterogenous systems, 
many applications, many users and applications which change 
over time by adding new versions.  IMHO the latter is the most
important benefit of a Modules environment.

> It doesn't seem to have made an impact in any FAQ (like
> comp.unix.shells for example) and indeed it took me a while
> to find the Tcl implementation using AltaVista.

The primary vehicle has been USENIX and LISA conferences.  LISA 
is particularly important because that is the Large Installation
System Administration conference.

> So where's the catch ?? 

The catch is that a single user system, like most Linuxes, don't
really need Modules: it is extra overhead.  Most large sites are
either flailing, using Modules, using something else like Depot 
or rolled their own.

The big question is: "what does the future hold?"  I've been 
mentally struggling with this for some time.  There was a time 
when a univeral GUI like CDE would be the answer.  With CDE there
is enough administrative flexibility to solve many of the problems
Modules solves for GUI environments.  But the new universal GUI is
really a browser.  And the web, especially with Java applets, has a
new way to solve many of the user environment and application 
distribution problems.  Alas, we still have the problem of non-GUI
environments, so there still isn't a clear winner.

 -- richard

The venerable John Furlani <jlf@...> said;

> Comments or other ideas on how to get the word out is appreciated...

I would suggest going about it like this;

1) Establish a WWW presence - even if it is simply one or two HTML files
   in ftp://ftp.eng.auburn.edu/pub/Modules/

   Keep it simple, like brief outline and abstract. Short installation
   requirements and a link to the ftp directory and/or files themselves.
   Ensure it has essential keywords; system administration, large
   installations, etc. Whatever you think people might search for.
   Cross reference USENIX - http://www.usenix.org/

   Mention the fact that Cray are using it with UNICOS (in at least
   one installation anyway). I have access to a few Cray sites
   but only one is currently adopting it. The modulefiles have
   definitely been written by Cray and not local sysadmins.

2) Point a few of the more popular WWW search engines at your main URL.

3) Announce the WWW page & package on a few key newsgroups.
   Keep it down to a screen full.
 
   Suggest: comp.os.linux.announce    comp.lang.tcl
            comp.unix.large           comp.unix.cray
            comp.unix.shells (?)      

4) See if you can get it mentioned in some of the FAQ's.
   It *is* in the comp.lang.tcl FAQ but this isn't sufficient IMHO.
 
5) Get members on this list to spread the word to a few appropriate lists. 
   This might take some co-ordination to avoid undue spamming ;-)

6) Get a few of the main system-admin user groups to mention it
   on their Web pages, circulars, mailing lists etc.

7) Establish a hypermail type archive for this list.

After all that just watch the ftp log files and your mail box fill up !
Be prepared to field lots of questions and persude people to join this list.

That's how I'd do it anyway. Others may have better ideas though.
I can certainly help with a little of the above if needed.

John P.

Let me add another perspective.  I work in an environment where we have
1500 SPARC workstations, literally hundreds of applications, etc.

Yet we don't use modules.

Why?  Because of inertia.

We have thousands of users - most of whom are quite inexperienced.
The system admins are working 50-60 hours a week already, just supporting
the current environment.  It's like Alice running in place as fast as she
can to stay in place.  To change things, or even just to start a new
with a new application, would take more time than they see benefits
on the return of investment.

What to do?  Here's some ideas.  Surely some of you use Linux, 4.4BSD,
FreeBSD or NetBSD.  Set up a Modules environment for your arena
using the default directory names as distributed with your OS.  Submit
it along with changes so that a new installation could make immediate
use of packages by use of the Modules.

Someone using Solaris build Module packages for each of the Solaris
install packages that are freely available, and talk to Ousterhout
to get him to promote the stuff inside of Sun as a cost savings to
the folk there.  Build a Solaris loadable package for Modules and submit
it to the freely distributable Solaris package archive.
Perhaps even share what you can of the various other packages that
have been built for common Solaris vendor packages like Frame, OpenWindows
X11R6, etc.  Oh - build some packages for the default Solaris 2.5 environment,
with a CDE package, SPARCworks, Java, etc.  Folks are going to start
moving to 2.5 faster and faster - if the environment is there, and advertised
as making it easier to upgrade, you will get their attention.
The easier it is for a system admin to 'plug and play' support for
things like Lotus, Excel, WABI, or whatever, the more likely they are
to use it - anything that makes their job easier is going to be considered.

HOWEVER - if Modules is seen as not being solid, as having weak points or
cause an increase in user problems, then you lose big time...
-- 
Larry W. Virden                 INET: lvirden@...
<URL:http://www.teraform.com/%7Elvirden/> is here! <*> O-
Unless explicitly stated to the contrary, nothing in this posting should 
be construed as representing my employer's opinions.

>4) See if you can get it mentioned in some of the FAQ's.
>   It *is* in the comp.lang.tcl FAQ but this isn't sufficient IMHO.

Do you mean that having it mentioned in my FAQ is not enough places,
or that what's in my FAQ is too small a reference?  If more info is needed
feel free to submit more information.

   Suggest: comp.os.linux.announce    comp.lang.tcl
            comp.unix.large           comp.unix.cray
            comp.unix.shells (?)      

I suggest that you try places like
comp.sys.sun.admin
comp.unix.admin

and perhaps these announce groups if you are announcing a new version

comp.sys.sun.announce
comp.unix.bsd.bsdi.announce
comp.unix.bsd.freebsd.announce
comp.unix.bsd.netbsd.announce
comp.unix.sco.announce
comp.lang.tcl.announce

Add the Mac and Windows environments if anyone has confirmed that
Modules works with Tcl 7.5 on those platforms.

Of course, things may need redesigned for the Mac I guess.  Too bad.


Again, as I mentioned in the last message, if someone wants to make
a splash, they should gather together a nice collection for several
of the most popular packages, using a naming structure appropriate for
the platform in question (/opt/package for Solaris, /usr/local/{bin,lib,man}
for most others I would guess).

Give folk a reason to use it.

Oh - and if this is all ready by July, make sure that someone attending
the Tcl/Tk workshop brings up the idea that extension writers supply
a Module interface for their packages!
-- 
Larry W. Virden                 INET: lvirden@...
<URL:http://www.teraform.com/%7Elvirden/> is here! <*> O-
Unless explicitly stated to the contrary, nothing in this posting should 
be construed as representing my employer's opinions.

In message <9606140604.AA23185@...>,
  Larry W. Virden, x2487 wrote:
[- snip -]
> HOWEVER - if Modules is seen as not being solid, as having weak points or
> cause an increase in user problems, then you lose big time...

For example, one of my "power users" that was helping to evaluate
modules complained that startup time was too slow.  He does a lot of
parallel work, and the time needed to start an rsh, including all of
the modules he uses, was obnoxious.  A way to cache your environment
would definitely help to address this problem.

Since I am not a tcl guru (yet), I havn't taken a look at this.  Does
anyone have any solutions or suggestions on this?

Brian

Brian T. Wightman                         Academic Computing, UW Oshkosh
wightman@...                                       800 Algoma Blvd
Phone: (414) 424-3020                                   Dempsey Hall 307
http://www.uwosh.edu/faculty_staff/wightman/    Oshkosh, Wisconsin 54901

> From: "Brian T. Wightman" <wightman@...>
> 
> In message <9606140604.AA23185@...>,
>   Larry W. Virden, x2487 wrote:
> [- snip -]
> > HOWEVER - if Modules is seen as not being solid, as having weak points or
> > cause an increase in user problems, then you lose big time...
> 
> For example, one of my "power users" that was helping to evaluate
> modules complained that startup time was too slow.  He does a lot of
> parallel work, and the time needed to start an rsh, including all of
> the modules he uses, was obnoxious.  A way to cache your environment
> would definitely help to address this problem.

At one time I did some extensive testing in this area.  The result is
that I now use the Korn shell.  What I found is that the way the
shell handles variables and paths is crucial to Modules performance.
The C-shell stinks when it comes to variables.  It is rather trivial to
do a test, something like:

	time modulecmd csh load xyz

will show how much time is really spent executing Modules.  The remainder
is consumed by your shell in the eval.

I originally did this testing in order to optimize my modulecmd build.
I tried various combinations of static and dynamic linking.  They were
inconclusive because modulecmd runs so fast.

Note: obviously one can write a modulefile that performs poorly.
I didn't :-)
 -- richard

> From: lvirden@... (Larry W. Virden, x2487)
> 
> Someone using Solaris build Module packages for each of the Solaris
> install packages that are freely available, and talk to Ousterhout
> to get him to promote the stuff inside of Sun as a cost savings to
> the folk there.

A few years back, John Furlani attempted to get Modules adopted
by SunIR (our internal IS group)  It didn't work, we use a home
grown Depot style setup which isn't nearly as nice.  Getting it
into something like base Solaris would be considerable effort.
Fortunately, there are better methods of getting Modules to the
masses...

>                  Build a Solaris loadable package for Modules and submit
> it to the freely distributable Solaris package archive.

I actually did that years ago when I was at Auburn.  There is a
package that loads /usr/modules to be the default setup complete
with init scripts.  The main modulefile directories were automounted
under /usr/local (== /opt/local)  The idea was that if /usr/modules
didn't exist, then the user would get a very basic path and an 
error message would be printed that the support staff understood.
Since I put the package in the jumpstart script, that never really
happened :-)  If the network or servers were down and /opt/local
wasn't available, they would still be left in a Modules environment
with a suitably default path (no network dependencies)  This allows
one to login and do *something* without being stuck because of a 
path dependency on /opt/local.

Anyway, packages are easy to do.  Perhaps someone still at Auburn
could share?

> Perhaps even share what you can of the various other packages that
> have been built for common Solaris vendor packages like Frame, OpenWindows
> X11R6, etc.  Oh - build some packages for the default Solaris 2.5 environment,
> with a CDE package, SPARCworks, Java, etc.  Folks are going to start
> moving to 2.5 faster and faster - if the environment is there, and advertised
> as making it easier to upgrade, you will get their attention.
> The easier it is for a system admin to 'plug and play' support for
> things like Lotus, Excel, WABI, or whatever, the more likely they are
> to use it - anything that makes their job easier is going to be considered.

Yes.  This could easily be rolled into a package.  Ideally, each
application would come with it's own Modulefile, but that is a pipedream.
Heck, many ISVs still don't use packages :-(
 -- richard

> From: "J. Pelan" <J.Pelan@...>

> 6) Get a few of the main system-admin user groups to mention it
>    on their Web pages, circulars, mailing lists etc.

USENIX and SAGE are good places to do this.  Anyone care to write
a short article?  I'm sure we could get it published in Login;

> 7) Establish a hypermail type archive for this list.

There is an archive.  But to my knowledge it's never been digested.
So it just grows and grows.  This list is handled by Majordomo and
should be able to reply with a digest.

 -- richard

Recently lvirden@... (Larry W. Virden, x2487) said;

me>4) See if you can get it mentioned in some of the FAQ's.
me>   It *is* in the comp.lang.tcl FAQ but this isn't sufficient IMHO.

lv> Do you mean that having it mentioned in my FAQ is not enough places,
lv> or that what's in my FAQ is too small a reference?  If more info is needed
lv> feel free to submit more information.

I meant that it should be mentioned in additional FAQ's, particularly
in the ones for the *.admin groups. If it were not for your entry in
the tcl FAQ I would not have found the modules implementation nor this
mailing list ;-)

Also relling@... (Richard Elling) said;

me> 7) Establish a hypermail type archive for this list.

re> There is an archive.  But to my knowledge it's never been digested.

I meant an archive that is accessible via the WWW, using a package
like hypermail or mhonarc. Such an archive is much easier to use
than a list server and exposes the contents to search engines as well
as being able to list messages by thread etc. I can provide such a service
however our server/network isn't even 90% reliable and as I'm UK based
that wouldn't suit many of you. However I'm willing to mirror the full
Modules site and hypermail archive, if & when one is established.

John P.

(Apologies if everyone gets this msg, or a variant thereof, twice)

I have tarred and gziped the package we use for auto-install
on Solaris machines.  This package contains a basic module
and the module command support files.

It is available at:
ftp://ftp.eng.auburn.edu/pub/Modules/contrib/AUengmod.tar.gz


Also there are old examples of our modules located at:
ftp://ftp.eng.auburn.edu/pub/Modules/contrib/auburn.modulefiles.tar.gz
These modulefiles are automounted to all of are clients.

On the hypermail issue... It is on my to-do list.  We have 
several mail list that would like hypermailed. 

If there is someone that wants to develop a web page for
modules and/or the contrib stuff I don't mind hosting
the web pages.  

--
Ray W. Hiltbrand                Ray.W.Hiltbrand@...
Engineering Network Services
Auburn University     http://www.eng.auburn.edu/~rayh/rayh.html
   If it doesn't do what you want, subclass and override.

>From the keyboard of Brian T. Wightman:
> 
> For example, one of my "power users" that was helping to evaluate
> modules complained that startup time was too slow.  He does a lot of
> parallel work, and the time needed to start an rsh, including all of
> the modules he uses, was obnoxious.  A way to cache your environment
> would definitely help to address this problem.

module load gnu X11/R6 frame 

is much much faster then

module load gnu
module load X11/R6
module load frame

because e.g. PATH will be set once, when using the first form, but
three times, when using the second form.

Maybe, this was the/a problem?

	--pwo
--
Avoid wearing, jewelry (including rings and chains), or other items that
could get caught in the chassis.  Avoid wearing or securely fasten loose
clothing, such as a tie, scarf, or sleeves.

	-- cisco, Catalyst 5000 Series Installation Guide, pg. 2-2
-- 
Peter W. Osel			Email:		pwo@...
Siemens AG - SBS DS 33		Phone:		+49 (89) 4144-3825
PO Box: 80 17 09		Fax:		+49 (89) 4144-2711
D-81617 Munich			Home:		pwo@...
Fed. Rep. of Germany		Home (Voice):	+49 (89) 613-6209
	<A HREF="http://www.ConnectDE.NET/~pwo/">My Home Page!</A>

Hi,

> > For example, one of my "power users" that was helping to evaluate
> > modules complained that startup time was too slow.  He does a lot of
> > parallel work, and the time needed to start an rsh, including all of
> > the modules he uses, was obnoxious.  A way to cache your environment
> > would definitely help to address this problem.

Perhaps You should change the makefile for generating optimized and
stripped binaries?

CFLAGS = -O3 instead of -g -O
and
INSTALL_PROGRAM = $(INSTALL) -s -m 0555
                             ^^ for stripping
reduced the sice of modulecmd by more than 1/2 (Sparc, gcc 2.7.0)
Perhaps this helps, when modulecmd is loaded over a network?

dpi

See <news:comp.unix.solaris> for article <32222217.5965@...>
which talks about modules.
-- 
Larry W. Virden                 INET: lvirden@...
<URL:http://www.teraform.com/%7Elvirden/> <*> O- "We are all Kosh."
Unless explicitly stated to the contrary, nothing in this posting should 
be construed as representing my employer's opinions.