|
From: Jonathan B. <jon...@gm...> - 2022-03-23 13:25:40
|
Hi Paul,
Thanks for the response! I took a look at this and spoke with a colleague,
but exit status is hard to grab from the module() method within python.
The only way we were able to pull this off was to ignore the suggested
python exec() statement found in the documentation and rather create our
own popen() statement. This allowed us to then check the status of that
process. What I don't like about this solution is that we're depending on
the originating shell to have properly sourced the module environment, so
that we can execute these 'os' commands. Here's a snippet of the code my
colleague came up with:
for mod in modules_to_test: proc = subprocess.run(f"module use
{mod_path};module load {mod}", stdout=subprocess.PIPE,
stderr=subprocess.PIPE, shell=True,
universal_newlines=True)
print("", "########", f"Test: {mod}", "########", sep="\n") #
Just FYI to see the attributes of the object # print(proc)
print(f"stdout: {proc.stdout}") print(f"stderr: {proc.stderr}")
print(f"return code: {proc.returncode}")
We Found that all output goes to proc.stdout rather than any errors going
to proc.stderr, regardless of a return code being 1 or 0. The return code
is the winner here. So, this works but I'm calling it a dirty solution
because I think that once I automate this and try to execute remotely on
our clusters, we'll lose the properly sourced environment.
On Tue, Mar 22, 2022 at 4:19 PM Paul Markfort <pau...@gm...> wrote:
> If you are just trying to, determine if there was an error thrown by the
> module command,
> Just test the exit status:
>
> EXIT STATUS
> The module command exits with 0 if its execution succeed.
> Otherwise 1
> is returned.
>
>
> Keep in mind that every call to the module command redirects sys.stdout so
> that the output is evaluated by python (so you probably can't capture that).
> And sys.stderr redirection by a module command might depend on what stderr
> is connected to.
>
> Last - stderr and stdout are buffered IO, so you may also have to do a
> flush before reading them.
>
>
> I should preface the below with, "I haven't coded in python":
>
> Maybe you need to test your code with no modules involved first.
> You might want to read this (seems like stringIO() may have some
> limitations).
>
> https://stackoverflow.com/questions/1218933/can-i-redirect-the-stdout-into-some-sort-of-string-buffer
>
>
>
>
>
>
>
> On 2022-03-22 09:18 AM, Jonathan Buck wrote:
> > Hi Jacques,
> >
> > Thanks for the quick reply! I have successfully use the module() command
> > to setup environments without issue. What I'm hoping to do is capture
> any
> > stderr messages to a variable. I have several use cases for this, most
> > importantly I'm attempting to automate checks against our hundreds of
> > modules to determine if any of them are broken when loaded. I'm unable
> to
> > redirect stdout/stderr in a manner that I've used before which I believe
> is
> > due to the command being a subprocess. I'm not a python guru, but the
> > popen statement declares " stdout=subprocess.PIPE" which I thought was
> the
> > right way to get the stdout returned to the originating process.
> >
> > All that being said, here's my test script:
> >
> > import os
> > import sys
> > from io import StringIO
> >
> > # Setup variables to hold stdout/stderr
> > result_stdout = StringIO()
> > result_stderr = StringIO()
> > sys.stdout = result_stdout
> > sys.stderr = result_stderr
> >
> > # setup environment module
> > exec(open('/p/app/Modules/4.7.1/init/python.py').read())
> >
> > # run
> > print("----- TRY RUNNING MODULE COMMAND -----")
> > module('--version')
> > print("----- DONE RUNNING MODULE COMMAND -----")
> > module_stdout = result_stdout.getvalue()
> > module_stderr = result_stderr.getvalue()
> >
> > # reset stdout/stderr to defaults
> > sys.stdout = sys.__stdout__
> > sys.stderr = sys.__stderr__
> >
> > # print out what was captured to the variables
> > print("===== TRY PRINTING STDOUT =====")
> > sys.stdout.write(module_stdout)
> > print("===== DONE PRINTING STDOUT =====")
> >
> > print("***** TRY PRINTING STDERR *****")
> > sys.stderr.write(module_stderr)
> > print("***** DONE PRINTING STDERR *****")
> >
> >
> > As you can see, it's simple. I make sure to reset stdout/stderr to
> > defaults, however these are my results:
> >
> > ~> python3 module_test.py
> > Modules Release 4.7.1 (2021-04-06)
> > ===== TRY PRINTING JSON STDOUT =====
> > ----- TRY RUNNING MODULE COMMAND -----
> > ----- DONE RUNNING MODULE COMMAND -----
> > ===== DONE PRINTING JSON STDOUT =====
> > ***** TRY PRINTING JSON STDERR *****
> > ***** DONE PRINTING JSON STDERR *****
> > ~>
> >
> > This shows that my stdout redirect is capturing the 'print' statements
> > for running the module command. However the output for the module
> > command prints at the very beginning and doesn't show up. Any
> > thoughts? I found a fancy way to do the same thing in bash, and I
> > proved that in bash the same output is going to stdout.
> >
> >
> > Very Respectfully,
> >
> > Jonathan
> >
> >
> > On Tue, Mar 22, 2022 at 8:06 AM Contact <dv...@cr...> wrote:
> >
> >> Hello,
> >>
> >> In my case I used modules from python with the following logic:
> >>
> >> import os
> >> exec(open(/home/jrp/modules/init/python.py')).read())
> >> module('avail')
> >> module('load', 'gcc/7')
> >> module('list')
> >>
> >> From the above example,
> >>
> >> - exec call will allow me to use modules from python itself and
> define
> >> module function
> >> - Each module function invocation will be redirected to module to
> >> perform required action
> >>
> >> As reminder, the content of stdout is used by modules to setup all
> >> required environment functions, functions/alias (if supported by current
> >> shell), ...
> >>
> >> I hope it will help you.
> >>
> >> Jacques
> >>
> >>
> >> Le 21/03/2022 à 20:09, Jonathan Buck a écrit :
> >>
> >> Hi,
> >>
> >> I'm attempting to capture and act on the stdout and stderr of the module
> >> command. The python implementation appears to be a subprocess for
> which I
> >> wasn't able to redirect with io.StringIO() as I have with other output
> >> capture.
> >>
> >> I attempted to look through the code, which I admit I don't fully
> >> understand and appears to have a one-liner style of executing the
> >> subprocess. I'm writing code that integrates with a production system,
> so
> >> I'm not able to modify the python.py code.
> >>
> >> Does anyone know how else this might be accomplished? Or should I make
> >> this a feature request?
> >>
> >> Thanks for your help,
> >> Jonathan
> >>
> >>
> >> _______________________________________________
> >> Modules-interest mailing listModules-interest
> @lists.sourceforge.nethttps://
> lists.sourceforge.net/lists/listinfo/modules-interest
> >>
> >>
> >
> >
> >
> > _______________________________________________
> > Modules-interest mailing list
> > Mod...@li...
> > https://lists.sourceforge.net/lists/listinfo/modules-interest
> >
>
> --
> --------------------------------------------------------
> The views and opinions expressed above are strictly
> those of the author(s). The content of this message has
> not been reviewed nor approved by any entity whatsoever.
> --------------------------------------------------------
> Paul FM Info: http://paulfm.com/~paulfm/
> --------------------------------------------------------
>
>
> _______________________________________________
> Modules-interest mailing list
> Mod...@li...
> https://lists.sourceforge.net/lists/listinfo/modules-interest
>
|
