|
From: Harlan S. <Har...@pf...> - 2002-01-13 09:50:37
|
Andy,
Before I left last night I thought I had this working - I made sure that the
ENV envariable gets set in the ...
I see - when you ssh in to a box you get a non-interactive shell and a ksh
script only runs the ENV file if it's already defined.
I suspect another choice would be to edit the basic environment setup
(possibly a change to sshd_conf) so the ENV variable gets set before
starting the ksh script. Either that or run:
ssh remotehost "env ENV=.foorc shellscript"
Longer term, I'd toss the ksh scripts and rewrite them in bash, or put the
modules init logic into a source-able shell script and suck it in at the top
of every script that needed it.
How many times have I said this rc stuff drives me crazy? And going thru
ssh adds another layer of complexity.
H
--
> Hello Harlan (and Dr. Owen)
>
> I did much more work over the night in our environment and have found some
> things which appear to work relatively well when running module commands in
> sub-shells.
>
> The initial problem we had (and which Dr. Owen alludes to below) is in the
> per-shell vs. per-login environment setup. While Dr. Owen's approach would
> work, in my case it wasn't feasible. (We have a lot of very specific ksh
> scripts that won't port without more work than I can do by Monday morning.)
>
> My basic goal was to:
> 1. Support the original ksh-specific scripts.
> 2. Use modules to control all environment settings.
> 3. Enable our prototype of a multi-platform background build system.
>
> So I had to find a workaround for modules that would do the following:
> 1. Use the work Harlan had done for us to handle multiple interactive-login
> shells (ksh, bash, csh, tcsh, etc.)
> 2. Account for non-interactive logins (e.g. via ssh) and for sub-shells
> (invoked scripts).
>
> Now for some items that I found out simply by trial-and-error while working
> on this issue, and that you both might find useful. For various historical
> (and bad) reasons, our development environments assume that assorted global
> commands exist in the shell namespace. For example, to do a production
> build, the developers (and the build environments!) assume that a magic
> "base_unit" command exists. Prior to the work Harlan did for me, we were
> sourcing in the scripts defining these commands as necessary.
>
> The first problem I have occurred when I was doing this simple command:
> ssh <local_host_name> -c base_unit
>
> This returned an error: "base_unit: not found". This was strange, since I
> had already defined base_unit in the login script. For example, doing the
> following sequence worked perfectly:
> ssh <local_host_name>
> base_unit
>
> It turns out that the difference lay in the fact that using the first syntax
> invokes a non-interactive shell. By making sure that the script file that
> ultimately defines "base_unit" was sourced in on a per-shell basis, I was
> able to fix the problem.
> Then I tried doing a manual call to our multi-platform background build
> script:
> ssh <local_host_name> -c /ic/build/m_build_all
>
> Once again, I got "base_unit: not found". This is where it got really ugly.
> It turns out that the "/ic/build/m_build_all" script is a ksh script. And,
> we use the "modulecmd" tool to set up the "base_unit" function, as in:
> ***CODE FROM A MODULES VERSION (TCL CODE)***
> # puts stderr "sh-like shell"
> set-alias base_unit "export DEV_ENV=baseline ; module load _ic"
>
> The problem got worse when I did the following:
> ssh <local_host_name> -c ksh
> base_unit
>
> The above worked! So, I was looking at a problem which worked fine with an
> interactive shell, but not with a sub-shell.
>
> At this point, I was getting somewhat desperate. I traced back through the
> configuration scripts for the login in question, and discovered that Harlan
> had already provided a nice hook for a ksh-specific sub-shell. (In our case,
> our login shell is always bash and the m_build_all script is ksh.) So I went
> into that hook, and did the following (*outside* of the "modulecmd" tool):
> ***CODE INSIDE A KSH-SCRIPT***
> base_unit() { export DEV_ENV=baseline; module load _ic }
>
> Then I ran:
> ssh <local_host_name> -c 'ksh -c base_unit'
>
> It worked, and I thought I had solved my problems. Then I tried the
> multi-platform build again:
> ssh <local_host_name> -c /ic/build/m_build_all
>
> And *still* received "base_unit: not found"! I finally resorted to reading
> ksh documentation and verified that my ENV var was set and valid (*and*
> being sourced correctly). I even verified that the base_unit definition was
> itself being sourced in (lots of ECHO statements by this point).
>
> As a last resort, I theorized that the alias wasn't getting exported to the
> global shell namespace. I saw that the "typeset" command would export
> functions, so I went into the ksh hook script, and converted the alias to a
> function and forced its export, as in:
> # HACK HACK HACK for ksh
> function base_unit {
> export DEV_ENV=baseline
> module load _ic
> }
> typeset -xf base_unit
>
> I then retried:
> ssh <local_host_name> -c /ic/build/m_build_all
>
> Success! I tried all other variations, and found success with them as well.
> And, for a final treat, I verified that the whole process worked the same on
> Cygwin (Windoze).
>
> I'm afraid beyond that flurry of trial-and-error I have nothing else to
> offer. The items that should be of interest to both of you have to do with
> the intricacies required: a) to make sure that appropriate environment
> scripts get sourced, accounting both for interactive and background shells;
> and, b) the problems that arose in trying to make functions/aliases defined
> using modules available to the same type of shells.
>
> It was a very long night, but it appears to work at last.
>
> Regards,
> Andy
>
>
> -----Original Message-----
> From: R.K. Owen Ph.D.
> To: Harlan Stenn
> Cc: Modules-Interest (E-mail); 'ha...@pf...'; Andy Bruce
> Sent: 1/12/2002 5:24 AM
> Subject: RE: shell scripts that need "modules" stuff
>
> Hello Harlan:
> Read through the INSTALL.RH7x document, because what it describes is
> how to set up modules for all users on a system (in this case
> Linux/RedHat 7.x).
> It also points out some of the pitfalls (especially with /bin/sh).
> Essentially, I have my system rc scripts alias sh to bash, because then
> you get the per-process sys rc scripts getting invoked to define the
> necessary modules functions, etc.
>
> Convert all your '#!/bin/sh' scripts to use '#!/bin/bash' (which should
> cause no problems). And for your scripts you can have regular module
> loads:
>
> if echo ${LOADEDMODULES:-none}| grep -q perl5 ; then
> : # OK
> else
> echo "perl5 module is not loaded ... don't waste my time!"
> echo "loading perl5 module myself!"
> module load perl5
> fi
>
> I think something like this is safer, because if each script invokes
> the modules/init scripts, then you have it rewriting the
> $HOME/.modulesbeginenv and, I think, resetting the *MODULE* env.vars.
> All you're really after is defining the aliases (or functions) for
> whatever shell you're in.
>
> R.K.
>
> +------------------+---------------------+
> | R.K.Owen,PhD | rk...@ow... |
> | KooZ Software | Environment Modules |
> | NERSC/LBNL | rk...@ne... |
> +------------------+---------------------+
>
> On Fri, 11 Jan 2002, Harlan Stenn wrote:
>
> > Here's more.
> >
> > We're writing scripts that want to call "commands" that are defined
> > by modulefiles.
> >
> > We noticed that sometimes these scripts worked, and sometimes they
> > didn't.
> >
> > I have just discovered that these scripts fail when they are called
> > by other scripts (as opposed from being invoked from the command
> > line) and this seems to be due to the fact that:
> >
> > - we're writing the scripts ASSUMING that the modules stuff is
> available
> > - - this has been true for us because we use shells that run startup
> (RC)
> > scripts on each shell invocation
> > - in performing some cleanup we are apparently making some changes
> that
> > mean that sometimes we don't run the entire suite of RC scripts
> > - this means that in some cases where scripts call scripts that the
> > shell aliases created by the modules RC stuff are not being created
> >
> > I figure there are two ways to proceed:
> >
> > - be VERY careful when writing scripts to make sure we source in a
> > suitable "modules init" function that also does our "loads"
> > - make sure we have per-shell (for *all* useful shells) RC files
> > that handle the modules init and our "loads"
> >
> > The first choice seems to be a bit better in that we'll only be doing
> the
> > modules setup on those scripts where we need them. The downside of
> this
> > method is that scripts will break until we find all of these cases.
> >
> > The second choice is less efficient (as there are going to be lots of
> > times where we won't be using modules but we have to do the work
> > anyway), but it has the benefit of putting the burden in a single
> place.
> >
> > Does this make sense?
> >
> > H
> >
> > -----Original Message-----
> > From: Harlan Stenn
> > Sent: Friday, January 11, 2002 7:05 PM
> > To: Modules-Interest (E-mail)
> > Subject: shell scripts that need "modules" stuff
> >
> >
> > So what's the right way to write shell (for arbitrary versions of
> shell)
> > script
> > that include "commands" and things that are defined by the modules
> stuff?
> >
> > Does each script that needs to do this have to source in
> > $MODULESHOME/init/$SHELL
> > (or something similar)?
> >
> > H
> >
|
