|
From: Xavier D. <xav...@gm...> - 2022-03-23 15:13:48
|
Hi Jonathan,
The exit status should be easy to grab from the module() python function.
This function basically returns this exit status as a boolean. I paste
below the current python code of the module() definition in python. It has
not changed a lot since Modules v4.1.
$ $MODULES_CMD python autoinit
import os
import re, subprocess
def module(*arguments):
ns = {}
exec(subprocess.Popen(['/usr/bin/tclsh',
'/usr/share/Modules/libexec/modulecmd.tcl', 'python'] + list(arguments),
stdout=subprocess.PIPE).communicate()[0], ns)
if '_mlstatus' in ns:
_mlstatus = ns['_mlstatus']
else:
_mlstatus = True
return _mlstatus
With the code example you gave, the module function used is the one defined
in a sh shell session (due to the shell=True parameter of subprocess.run).
Usually the module shell function returns the output on stdout (stderr
messages are redirected to stdout after the evaluation of the shell code
generated on stdout).
Regards,
Xavier
Le mer. 23 mars 2022 à 14:26, Jonathan Buck <jon...@gm...> a
écrit :
> Hi Paul,
>
> Thanks for the response! I took a look at this and spoke with a
> colleague, but exit status is hard to grab from the module() method within
> python. The only way we were able to pull this off was to ignore the
> suggested python exec() statement found in the documentation and rather
> create our own popen() statement. This allowed us to then check the status
> of that process. What I don't like about this solution is that we're
> depending on the originating shell to have properly sourced the module
> environment, so that we can execute these 'os' commands. Here's a snippet
> of the code my colleague came up with:
>
> for mod in modules_to_test: proc = subprocess.run(f"module use {mod_path};module load {mod}", stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True, universal_newlines=True)
> print("", "########", f"Test: {mod}", "########", sep="\n") # Just FYI to see the attributes of the object # print(proc) print(f"stdout: {proc.stdout}") print(f"stderr: {proc.stderr}") print(f"return code: {proc.returncode}")
>
> We Found that all output goes to proc.stdout rather than any errors going
> to proc.stderr, regardless of a return code being 1 or 0. The return code
> is the winner here. So, this works but I'm calling it a dirty solution
> because I think that once I automate this and try to execute remotely on
> our clusters, we'll lose the properly sourced environment.
>
> On Tue, Mar 22, 2022 at 4:19 PM Paul Markfort <pau...@gm...> wrote:
>
>> If you are just trying to, determine if there was an error thrown by the
>> module command,
>> Just test the exit status:
>>
>> EXIT STATUS
>> The module command exits with 0 if its execution succeed.
>> Otherwise 1
>> is returned.
>>
>>
>> Keep in mind that every call to the module command redirects sys.stdout
>> so that the output is evaluated by python (so you probably can't capture
>> that).
>> And sys.stderr redirection by a module command might depend on what
>> stderr is connected to.
>>
>> Last - stderr and stdout are buffered IO, so you may also have to do a
>> flush before reading them.
>>
>>
>> I should preface the below with, "I haven't coded in python":
>>
>> Maybe you need to test your code with no modules involved first.
>> You might want to read this (seems like stringIO() may have some
>> limitations).
>>
>> https://stackoverflow.com/questions/1218933/can-i-redirect-the-stdout-into-some-sort-of-string-buffer
>>
>>
>>
>>
>>
>>
>>
>> On 2022-03-22 09:18 AM, Jonathan Buck wrote:
>> > Hi Jacques,
>> >
>> > Thanks for the quick reply! I have successfully use the module()
>> command
>> > to setup environments without issue. What I'm hoping to do is capture
>> any
>> > stderr messages to a variable. I have several use cases for this, most
>> > importantly I'm attempting to automate checks against our hundreds of
>> > modules to determine if any of them are broken when loaded. I'm unable
>> to
>> > redirect stdout/stderr in a manner that I've used before which I
>> believe is
>> > due to the command being a subprocess. I'm not a python guru, but the
>> > popen statement declares " stdout=subprocess.PIPE" which I thought was
>> the
>> > right way to get the stdout returned to the originating process.
>> >
>> > All that being said, here's my test script:
>> >
>> > import os
>> > import sys
>> > from io import StringIO
>> >
>> > # Setup variables to hold stdout/stderr
>> > result_stdout = StringIO()
>> > result_stderr = StringIO()
>> > sys.stdout = result_stdout
>> > sys.stderr = result_stderr
>> >
>> > # setup environment module
>> > exec(open('/p/app/Modules/4.7.1/init/python.py').read())
>> >
>> > # run
>> > print("----- TRY RUNNING MODULE COMMAND -----")
>> > module('--version')
>> > print("----- DONE RUNNING MODULE COMMAND -----")
>> > module_stdout = result_stdout.getvalue()
>> > module_stderr = result_stderr.getvalue()
>> >
>> > # reset stdout/stderr to defaults
>> > sys.stdout = sys.__stdout__
>> > sys.stderr = sys.__stderr__
>> >
>> > # print out what was captured to the variables
>> > print("===== TRY PRINTING STDOUT =====")
>> > sys.stdout.write(module_stdout)
>> > print("===== DONE PRINTING STDOUT =====")
>> >
>> > print("***** TRY PRINTING STDERR *****")
>> > sys.stderr.write(module_stderr)
>> > print("***** DONE PRINTING STDERR *****")
>> >
>> >
>> > As you can see, it's simple. I make sure to reset stdout/stderr to
>> > defaults, however these are my results:
>> >
>> > ~> python3 module_test.py
>> > Modules Release 4.7.1 (2021-04-06)
>> > ===== TRY PRINTING JSON STDOUT =====
>> > ----- TRY RUNNING MODULE COMMAND -----
>> > ----- DONE RUNNING MODULE COMMAND -----
>> > ===== DONE PRINTING JSON STDOUT =====
>> > ***** TRY PRINTING JSON STDERR *****
>> > ***** DONE PRINTING JSON STDERR *****
>> > ~>
>> >
>> > This shows that my stdout redirect is capturing the 'print' statements
>> > for running the module command. However the output for the module
>> > command prints at the very beginning and doesn't show up. Any
>> > thoughts? I found a fancy way to do the same thing in bash, and I
>> > proved that in bash the same output is going to stdout.
>> >
>> >
>> > Very Respectfully,
>> >
>> > Jonathan
>> >
>> >
>> > On Tue, Mar 22, 2022 at 8:06 AM Contact <dv...@cr...> wrote:
>> >
>> >> Hello,
>> >>
>> >> In my case I used modules from python with the following logic:
>> >>
>> >> import os
>> >> exec(open(/home/jrp/modules/init/python.py')).read())
>> >> module('avail')
>> >> module('load', 'gcc/7')
>> >> module('list')
>> >>
>> >> From the above example,
>> >>
>> >> - exec call will allow me to use modules from python itself and
>> define
>> >> module function
>> >> - Each module function invocation will be redirected to module to
>> >> perform required action
>> >>
>> >> As reminder, the content of stdout is used by modules to setup all
>> >> required environment functions, functions/alias (if supported by
>> current
>> >> shell), ...
>> >>
>> >> I hope it will help you.
>> >>
>> >> Jacques
>> >>
>> >>
>> >> Le 21/03/2022 à 20:09, Jonathan Buck a écrit :
>> >>
>> >> Hi,
>> >>
>> >> I'm attempting to capture and act on the stdout and stderr of the
>> module
>> >> command. The python implementation appears to be a subprocess for
>> which I
>> >> wasn't able to redirect with io.StringIO() as I have with other output
>> >> capture.
>> >>
>> >> I attempted to look through the code, which I admit I don't fully
>> >> understand and appears to have a one-liner style of executing the
>> >> subprocess. I'm writing code that integrates with a production
>> system, so
>> >> I'm not able to modify the python.py code.
>> >>
>> >> Does anyone know how else this might be accomplished? Or should I make
>> >> this a feature request?
>> >>
>> >> Thanks for your help,
>> >> Jonathan
>> >>
>> >>
>> >> _______________________________________________
>> >> Modules-interest mailing listModules-interest
>> @lists.sourceforge.nethttps://
>> lists.sourceforge.net/lists/listinfo/modules-interest
>> >>
>> >>
>> >
>> >
>> >
>> > _______________________________________________
>> > Modules-interest mailing list
>> > Mod...@li...
>> > https://lists.sourceforge.net/lists/listinfo/modules-interest
>> >
>>
>> --
>> --------------------------------------------------------
>> The views and opinions expressed above are strictly
>> those of the author(s). The content of this message has
>> not been reviewed nor approved by any entity whatsoever.
>> --------------------------------------------------------
>> Paul FM Info: http://paulfm.com/~paulfm/
>> --------------------------------------------------------
>>
>>
>> _______________________________________________
>> Modules-interest mailing list
>> Mod...@li...
>> https://lists.sourceforge.net/lists/listinfo/modules-interest
>>
> _______________________________________________
> Modules-interest mailing list
> Mod...@li...
> https://lists.sourceforge.net/lists/listinfo/modules-interest
>
|
