|
From: Xavier D. <xav...@gm...> - 2022-03-23 15:46:50
|
Le mer. 23 mars 2022 à 16:31, Jonathan Buck <jon...@gm...> a
écrit :
> Hi Xavier,
>
> This is great! Thanks for showing me how to expose the underlying
> function! One thing to note is that everything returned comes through
> stderr. So, as you were saying, I wasn't able to trap anything because
> everything was coming through stderr.
>
> In my testing, _mlstatus is returned by stdout. Are there other values
> that might be returned by stdout other than "_mlstatus = False" ?
>
> Am I right in assuming '_mlstatus' is the return code of the tcl execution?
>
_mlstatus is the variable that hold the return code of the evaluation. On
stdout also goes all the environment changes implied by modulefile
evaluation.
> The only problem I still have is that I can't seem to capture that stderr
> output to a variable. Currently, it goes to the console, no matter what I
> do. I'm going to play around with it and see if this is because of the
> exec() or the popen() statements. Ideally, I would like to capture the
> output when the _mlstatus is False. For my use case, This feature would
> allow me to compile a list of my 'bad' modules and their associated output.
>
> I've been playing with your initial code this morning, teasing it apart
> and designing a return object containing the output and the return code.
> This might not be the preferred methodology, but either way I'll report
> back with what I find.
>
With the code I proposed this morning, you should be able to get the
content from sys.stderr. On my side, I have run this proposed code in the
python script you provided and I correctly got that stderr output between
the TRY/DONE PRINTING STDERR lines.
Regards,
Xavier
> Thanks again and with all respect,
> Jonathan
>
>
> On Wed, Mar 23, 2022 at 11:14 AM Xavier Delaruelle <
> xav...@gm...> wrote:
>
>> Hi Jonathan,
>>
>> The exit status should be easy to grab from the module() python function.
>> This function basically returns this exit status as a boolean. I paste
>> below the current python code of the module() definition in python. It has
>> not changed a lot since Modules v4.1.
>>
>> $ $MODULES_CMD python autoinit
>> import os
>> import re, subprocess
>> def module(*arguments):
>> ns = {}
>> exec(subprocess.Popen(['/usr/bin/tclsh',
>> '/usr/share/Modules/libexec/modulecmd.tcl', 'python'] + list(arguments),
>> stdout=subprocess.PIPE).communicate()[0], ns)
>> if '_mlstatus' in ns:
>> _mlstatus = ns['_mlstatus']
>> else:
>> _mlstatus = True
>> return _mlstatus
>>
>> With the code example you gave, the module function used is the one
>> defined in a sh shell session (due to the shell=True parameter of
>> subprocess.run). Usually the module shell function returns the output on
>> stdout (stderr messages are redirected to stdout after the evaluation of
>> the shell code generated on stdout).
>>
>> Regards,
>> Xavier
>>
>> Le mer. 23 mars 2022 à 14:26, Jonathan Buck <jon...@gm...> a
>> écrit :
>>
>>> Hi Paul,
>>>
>>> Thanks for the response! I took a look at this and spoke with a
>>> colleague, but exit status is hard to grab from the module() method within
>>> python. The only way we were able to pull this off was to ignore the
>>> suggested python exec() statement found in the documentation and rather
>>> create our own popen() statement. This allowed us to then check the status
>>> of that process. What I don't like about this solution is that we're
>>> depending on the originating shell to have properly sourced the module
>>> environment, so that we can execute these 'os' commands. Here's a snippet
>>> of the code my colleague came up with:
>>>
>>> for mod in modules_to_test: proc = subprocess.run(f"module use {mod_path};module load {mod}", stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True, universal_newlines=True)
>>> print("", "########", f"Test: {mod}", "########", sep="\n") # Just FYI to see the attributes of the object # print(proc) print(f"stdout: {proc.stdout}") print(f"stderr: {proc.stderr}") print(f"return code: {proc.returncode}")
>>>
>>> We Found that all output goes to proc.stdout rather than any errors
>>> going to proc.stderr, regardless of a return code being 1 or 0. The return
>>> code is the winner here. So, this works but I'm calling it a dirty solution
>>> because I think that once I automate this and try to execute remotely on
>>> our clusters, we'll lose the properly sourced environment.
>>>
>>> On Tue, Mar 22, 2022 at 4:19 PM Paul Markfort <pau...@gm...>
>>> wrote:
>>>
>>>> If you are just trying to, determine if there was an error thrown by
>>>> the module command,
>>>> Just test the exit status:
>>>>
>>>> EXIT STATUS
>>>> The module command exits with 0 if its execution succeed.
>>>> Otherwise 1
>>>> is returned.
>>>>
>>>>
>>>> Keep in mind that every call to the module command redirects sys.stdout
>>>> so that the output is evaluated by python (so you probably can't capture
>>>> that).
>>>> And sys.stderr redirection by a module command might depend on what
>>>> stderr is connected to.
>>>>
>>>> Last - stderr and stdout are buffered IO, so you may also have to do a
>>>> flush before reading them.
>>>>
>>>>
>>>> I should preface the below with, "I haven't coded in python":
>>>>
>>>> Maybe you need to test your code with no modules involved first.
>>>> You might want to read this (seems like stringIO() may have some
>>>> limitations).
>>>>
>>>> https://stackoverflow.com/questions/1218933/can-i-redirect-the-stdout-into-some-sort-of-string-buffer
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On 2022-03-22 09:18 AM, Jonathan Buck wrote:
>>>> > Hi Jacques,
>>>> >
>>>> > Thanks for the quick reply! I have successfully use the module()
>>>> command
>>>> > to setup environments without issue. What I'm hoping to do is
>>>> capture any
>>>> > stderr messages to a variable. I have several use cases for this,
>>>> most
>>>> > importantly I'm attempting to automate checks against our hundreds of
>>>> > modules to determine if any of them are broken when loaded. I'm
>>>> unable to
>>>> > redirect stdout/stderr in a manner that I've used before which I
>>>> believe is
>>>> > due to the command being a subprocess. I'm not a python guru, but the
>>>> > popen statement declares " stdout=subprocess.PIPE" which I thought
>>>> was the
>>>> > right way to get the stdout returned to the originating process.
>>>> >
>>>> > All that being said, here's my test script:
>>>> >
>>>> > import os
>>>> > import sys
>>>> > from io import StringIO
>>>> >
>>>> > # Setup variables to hold stdout/stderr
>>>> > result_stdout = StringIO()
>>>> > result_stderr = StringIO()
>>>> > sys.stdout = result_stdout
>>>> > sys.stderr = result_stderr
>>>> >
>>>> > # setup environment module
>>>> > exec(open('/p/app/Modules/4.7.1/init/python.py').read())
>>>> >
>>>> > # run
>>>> > print("----- TRY RUNNING MODULE COMMAND -----")
>>>> > module('--version')
>>>> > print("----- DONE RUNNING MODULE COMMAND -----")
>>>> > module_stdout = result_stdout.getvalue()
>>>> > module_stderr = result_stderr.getvalue()
>>>> >
>>>> > # reset stdout/stderr to defaults
>>>> > sys.stdout = sys.__stdout__
>>>> > sys.stderr = sys.__stderr__
>>>> >
>>>> > # print out what was captured to the variables
>>>> > print("===== TRY PRINTING STDOUT =====")
>>>> > sys.stdout.write(module_stdout)
>>>> > print("===== DONE PRINTING STDOUT =====")
>>>> >
>>>> > print("***** TRY PRINTING STDERR *****")
>>>> > sys.stderr.write(module_stderr)
>>>> > print("***** DONE PRINTING STDERR *****")
>>>> >
>>>> >
>>>> > As you can see, it's simple. I make sure to reset stdout/stderr to
>>>> > defaults, however these are my results:
>>>> >
>>>> > ~> python3 module_test.py
>>>> > Modules Release 4.7.1 (2021-04-06)
>>>> > ===== TRY PRINTING JSON STDOUT =====
>>>> > ----- TRY RUNNING MODULE COMMAND -----
>>>> > ----- DONE RUNNING MODULE COMMAND -----
>>>> > ===== DONE PRINTING JSON STDOUT =====
>>>> > ***** TRY PRINTING JSON STDERR *****
>>>> > ***** DONE PRINTING JSON STDERR *****
>>>> > ~>
>>>> >
>>>> > This shows that my stdout redirect is capturing the 'print' statements
>>>> > for running the module command. However the output for the module
>>>> > command prints at the very beginning and doesn't show up. Any
>>>> > thoughts? I found a fancy way to do the same thing in bash, and I
>>>> > proved that in bash the same output is going to stdout.
>>>> >
>>>> >
>>>> > Very Respectfully,
>>>> >
>>>> > Jonathan
>>>> >
>>>> >
>>>> > On Tue, Mar 22, 2022 at 8:06 AM Contact <dv...@cr...> wrote:
>>>> >
>>>> >> Hello,
>>>> >>
>>>> >> In my case I used modules from python with the following logic:
>>>> >>
>>>> >> import os
>>>> >> exec(open(/home/jrp/modules/init/python.py')).read())
>>>> >> module('avail')
>>>> >> module('load', 'gcc/7')
>>>> >> module('list')
>>>> >>
>>>> >> From the above example,
>>>> >>
>>>> >> - exec call will allow me to use modules from python itself and
>>>> define
>>>> >> module function
>>>> >> - Each module function invocation will be redirected to module to
>>>> >> perform required action
>>>> >>
>>>> >> As reminder, the content of stdout is used by modules to setup all
>>>> >> required environment functions, functions/alias (if supported by
>>>> current
>>>> >> shell), ...
>>>> >>
>>>> >> I hope it will help you.
>>>> >>
>>>> >> Jacques
>>>> >>
>>>> >>
>>>> >> Le 21/03/2022 à 20:09, Jonathan Buck a écrit :
>>>> >>
>>>> >> Hi,
>>>> >>
>>>> >> I'm attempting to capture and act on the stdout and stderr of the
>>>> module
>>>> >> command. The python implementation appears to be a subprocess for
>>>> which I
>>>> >> wasn't able to redirect with io.StringIO() as I have with other
>>>> output
>>>> >> capture.
>>>> >>
>>>> >> I attempted to look through the code, which I admit I don't fully
>>>> >> understand and appears to have a one-liner style of executing the
>>>> >> subprocess. I'm writing code that integrates with a production
>>>> system, so
>>>> >> I'm not able to modify the python.py code.
>>>> >>
>>>> >> Does anyone know how else this might be accomplished? Or should I
>>>> make
>>>> >> this a feature request?
>>>> >>
>>>> >> Thanks for your help,
>>>> >> Jonathan
>>>>
>>>
|
