|
From: Jonathan B. <jon...@gm...> - 2022-03-23 16:20:23
|
Hi Xavier,
Thanks for the additional information about _mlstatus. I tried to
implement the function as suggested, and I did not have the same success as
you. I may have done this completely wrong. I placed your function inside
of my own script, so that it's defined locally. I'm assuming that may be
the only difference between us, however I'm not sure why that would be a
problem. For what it's worth, my environment is:
bash version 4.4.23
python version 3.6.15
module version 4.7.1
#!/usr/bin/python3
import os
import sys
from io import StringIO
import re, subprocess
def module(*arguments):
ns = {}
exec(subprocess.Popen(['/usr/bin/tclsh',
'/usr/share/Modules/libexec/modulecmd.tcl', 'python'] + list(arguments),
stdout=subprocess.PIPE).communicate()[0], ns)
if '_mlstatus' in ns:
_mlstatus = ns['_mlstatus']
else:
_mlstatus = True
return _mlstatus
# Setup variables to hold stdout/stderr
result_stdout = StringIO()
result_stderr = StringIO()
sys.stdout = result_stdout
sys.stderr = result_stderr
# run
print("----- TRY RUNNING MODULE COMMAND -----")
module('--version')
print("----- DONE RUNNING MODULE COMMAND -----")
module_stdout = result_stdout.getvalue()
module_stderr = result_stderr.getvalue()
# reset stdout/stderr
sys.stdout = sys.__stdout__
sys.stderr = sys.__stderr__
# print out what was captured to the variables
print("===== TRY PRINTING STDOUT =====")
sys.stdout.write(module_stdout)
print("===== DONE PRINTING STDOUT =====")
print("***** TRY PRINTING STDERR *****")
sys.stdout.write(module_stderr)
print("***** DONE PRINTING STDERR *****")
The result of that script is:
~> python3 module_test.py
Modules Release 4.7.1 (2021-04-06)
===== TRY PRINTING STDOUT =====
----- TRY RUNNING MODULE COMMAND -----
----- DONE RUNNING MODULE COMMAND -----
===== DONE PRINTING STDOUT =====
***** TRY PRINTING STDERR *****
***** DONE PRINTING STDERR *****
On Wed, Mar 23, 2022 at 11:47 AM Xavier Delaruelle <
xav...@gm...> wrote:
>
> Le mer. 23 mars 2022 à 16:31, Jonathan Buck <jon...@gm...> a
> écrit :
>
>> Hi Xavier,
>>
>> This is great! Thanks for showing me how to expose the underlying
>> function! One thing to note is that everything returned comes through
>> stderr. So, as you were saying, I wasn't able to trap anything because
>> everything was coming through stderr.
>>
>> In my testing, _mlstatus is returned by stdout. Are there other values
>> that might be returned by stdout other than "_mlstatus = False" ?
>>
>> Am I right in assuming '_mlstatus' is the return code of the tcl
>> execution?
>>
>
> _mlstatus is the variable that hold the return code of the evaluation. On
> stdout also goes all the environment changes implied by modulefile
> evaluation.
>
>
>> The only problem I still have is that I can't seem to capture that stderr
>> output to a variable. Currently, it goes to the console, no matter what I
>> do. I'm going to play around with it and see if this is because of the
>> exec() or the popen() statements. Ideally, I would like to capture the
>> output when the _mlstatus is False. For my use case, This feature would
>> allow me to compile a list of my 'bad' modules and their associated output.
>>
>> I've been playing with your initial code this morning, teasing it apart
>> and designing a return object containing the output and the return code.
>> This might not be the preferred methodology, but either way I'll report
>> back with what I find.
>>
>
> With the code I proposed this morning, you should be able to get the
> content from sys.stderr. On my side, I have run this proposed code in the
> python script you provided and I correctly got that stderr output between
> the TRY/DONE PRINTING STDERR lines.
>
> Regards,
> Xavier
>
>
>
>> Thanks again and with all respect,
>> Jonathan
>>
>>
>> On Wed, Mar 23, 2022 at 11:14 AM Xavier Delaruelle <
>> xav...@gm...> wrote:
>>
>>> Hi Jonathan,
>>>
>>> The exit status should be easy to grab from the module() python
>>> function. This function basically returns this exit status as a boolean. I
>>> paste below the current python code of the module() definition in python.
>>> It has not changed a lot since Modules v4.1.
>>>
>>> $ $MODULES_CMD python autoinit
>>> import os
>>> import re, subprocess
>>> def module(*arguments):
>>> ns = {}
>>> exec(subprocess.Popen(['/usr/bin/tclsh',
>>> '/usr/share/Modules/libexec/modulecmd.tcl', 'python'] + list(arguments),
>>> stdout=subprocess.PIPE).communicate()[0], ns)
>>> if '_mlstatus' in ns:
>>> _mlstatus = ns['_mlstatus']
>>> else:
>>> _mlstatus = True
>>> return _mlstatus
>>>
>>> With the code example you gave, the module function used is the one
>>> defined in a sh shell session (due to the shell=True parameter of
>>> subprocess.run). Usually the module shell function returns the output on
>>> stdout (stderr messages are redirected to stdout after the evaluation of
>>> the shell code generated on stdout).
>>>
>>> Regards,
>>> Xavier
>>>
>>> Le mer. 23 mars 2022 à 14:26, Jonathan Buck <jon...@gm...> a
>>> écrit :
>>>
>>>> Hi Paul,
>>>>
>>>> Thanks for the response! I took a look at this and spoke with a
>>>> colleague, but exit status is hard to grab from the module() method within
>>>> python. The only way we were able to pull this off was to ignore the
>>>> suggested python exec() statement found in the documentation and rather
>>>> create our own popen() statement. This allowed us to then check the status
>>>> of that process. What I don't like about this solution is that we're
>>>> depending on the originating shell to have properly sourced the module
>>>> environment, so that we can execute these 'os' commands. Here's a snippet
>>>> of the code my colleague came up with:
>>>>
>>>> for mod in modules_to_test: proc = subprocess.run(f"module use {mod_path};module load {mod}", stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True, universal_newlines=True)
>>>> print("", "########", f"Test: {mod}", "########", sep="\n") # Just FYI to see the attributes of the object # print(proc) print(f"stdout: {proc.stdout}") print(f"stderr: {proc.stderr}") print(f"return code: {proc.returncode}")
>>>>
>>>> We Found that all output goes to proc.stdout rather than any errors
>>>> going to proc.stderr, regardless of a return code being 1 or 0. The return
>>>> code is the winner here. So, this works but I'm calling it a dirty solution
>>>> because I think that once I automate this and try to execute remotely on
>>>> our clusters, we'll lose the properly sourced environment.
>>>>
>>>> On Tue, Mar 22, 2022 at 4:19 PM Paul Markfort <pau...@gm...>
>>>> wrote:
>>>>
>>>>> If you are just trying to, determine if there was an error thrown by
>>>>> the module command,
>>>>> Just test the exit status:
>>>>>
>>>>> EXIT STATUS
>>>>> The module command exits with 0 if its execution succeed.
>>>>> Otherwise 1
>>>>> is returned.
>>>>>
>>>>>
>>>>> Keep in mind that every call to the module command redirects
>>>>> sys.stdout so that the output is evaluated by python (so you probably can't
>>>>> capture that).
>>>>> And sys.stderr redirection by a module command might depend on what
>>>>> stderr is connected to.
>>>>>
>>>>> Last - stderr and stdout are buffered IO, so you may also have to do a
>>>>> flush before reading them.
>>>>>
>>>>>
>>>>> I should preface the below with, "I haven't coded in python":
>>>>>
>>>>> Maybe you need to test your code with no modules involved first.
>>>>> You might want to read this (seems like stringIO() may have some
>>>>> limitations).
>>>>>
>>>>> https://stackoverflow.com/questions/1218933/can-i-redirect-the-stdout-into-some-sort-of-string-buffer
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 2022-03-22 09:18 AM, Jonathan Buck wrote:
>>>>> > Hi Jacques,
>>>>> >
>>>>> > Thanks for the quick reply! I have successfully use the module()
>>>>> command
>>>>> > to setup environments without issue. What I'm hoping to do is
>>>>> capture any
>>>>> > stderr messages to a variable. I have several use cases for this,
>>>>> most
>>>>> > importantly I'm attempting to automate checks against our hundreds of
>>>>> > modules to determine if any of them are broken when loaded. I'm
>>>>> unable to
>>>>> > redirect stdout/stderr in a manner that I've used before which I
>>>>> believe is
>>>>> > due to the command being a subprocess. I'm not a python guru, but
>>>>> the
>>>>> > popen statement declares " stdout=subprocess.PIPE" which I thought
>>>>> was the
>>>>> > right way to get the stdout returned to the originating process.
>>>>> >
>>>>> > All that being said, here's my test script:
>>>>> >
>>>>> > import os
>>>>> > import sys
>>>>> > from io import StringIO
>>>>> >
>>>>> > # Setup variables to hold stdout/stderr
>>>>> > result_stdout = StringIO()
>>>>> > result_stderr = StringIO()
>>>>> > sys.stdout = result_stdout
>>>>> > sys.stderr = result_stderr
>>>>> >
>>>>> > # setup environment module
>>>>> > exec(open('/p/app/Modules/4.7.1/init/python.py').read())
>>>>> >
>>>>> > # run
>>>>> > print("----- TRY RUNNING MODULE COMMAND -----")
>>>>> > module('--version')
>>>>> > print("----- DONE RUNNING MODULE COMMAND -----")
>>>>> > module_stdout = result_stdout.getvalue()
>>>>> > module_stderr = result_stderr.getvalue()
>>>>> >
>>>>> > # reset stdout/stderr to defaults
>>>>> > sys.stdout = sys.__stdout__
>>>>> > sys.stderr = sys.__stderr__
>>>>> >
>>>>> > # print out what was captured to the variables
>>>>> > print("===== TRY PRINTING STDOUT =====")
>>>>> > sys.stdout.write(module_stdout)
>>>>> > print("===== DONE PRINTING STDOUT =====")
>>>>> >
>>>>> > print("***** TRY PRINTING STDERR *****")
>>>>> > sys.stderr.write(module_stderr)
>>>>> > print("***** DONE PRINTING STDERR *****")
>>>>> >
>>>>> >
>>>>> > As you can see, it's simple. I make sure to reset stdout/stderr to
>>>>> > defaults, however these are my results:
>>>>> >
>>>>> > ~> python3 module_test.py
>>>>> > Modules Release 4.7.1 (2021-04-06)
>>>>> > ===== TRY PRINTING JSON STDOUT =====
>>>>> > ----- TRY RUNNING MODULE COMMAND -----
>>>>> > ----- DONE RUNNING MODULE COMMAND -----
>>>>> > ===== DONE PRINTING JSON STDOUT =====
>>>>> > ***** TRY PRINTING JSON STDERR *****
>>>>> > ***** DONE PRINTING JSON STDERR *****
>>>>> > ~>
>>>>> >
>>>>> > This shows that my stdout redirect is capturing the 'print'
>>>>> statements
>>>>> > for running the module command. However the output for the module
>>>>> > command prints at the very beginning and doesn't show up. Any
>>>>> > thoughts? I found a fancy way to do the same thing in bash, and I
>>>>> > proved that in bash the same output is going to stdout.
>>>>> >
>>>>> >
>>>>> > Very Respectfully,
>>>>> >
>>>>> > Jonathan
>>>>> >
>>>>> >
>>>>> > On Tue, Mar 22, 2022 at 8:06 AM Contact <dv...@cr...> wrote:
>>>>> >
>>>>> >> Hello,
>>>>> >>
>>>>> >> In my case I used modules from python with the following logic:
>>>>> >>
>>>>> >> import os
>>>>> >> exec(open(/home/jrp/modules/init/python.py')).read())
>>>>> >> module('avail')
>>>>> >> module('load', 'gcc/7')
>>>>> >> module('list')
>>>>> >>
>>>>> >> From the above example,
>>>>> >>
>>>>> >> - exec call will allow me to use modules from python itself and
>>>>> define
>>>>> >> module function
>>>>> >> - Each module function invocation will be redirected to module
>>>>> to
>>>>> >> perform required action
>>>>> >>
>>>>> >> As reminder, the content of stdout is used by modules to setup all
>>>>> >> required environment functions, functions/alias (if supported by
>>>>> current
>>>>> >> shell), ...
>>>>> >>
>>>>> >> I hope it will help you.
>>>>> >>
>>>>> >> Jacques
>>>>> >>
>>>>> >>
>>>>> >> Le 21/03/2022 à 20:09, Jonathan Buck a écrit :
>>>>> >>
>>>>> >> Hi,
>>>>> >>
>>>>> >> I'm attempting to capture and act on the stdout and stderr of the
>>>>> module
>>>>> >> command. The python implementation appears to be a subprocess for
>>>>> which I
>>>>> >> wasn't able to redirect with io.StringIO() as I have with other
>>>>> output
>>>>> >> capture.
>>>>> >>
>>>>> >> I attempted to look through the code, which I admit I don't fully
>>>>> >> understand and appears to have a one-liner style of executing the
>>>>> >> subprocess. I'm writing code that integrates with a production
>>>>> system, so
>>>>> >> I'm not able to modify the python.py code.
>>>>> >>
>>>>> >> Does anyone know how else this might be accomplished? Or should I
>>>>> make
>>>>> >> this a feature request?
>>>>> >>
>>>>> >> Thanks for your help,
>>>>> >> Jonathan
>>>>>
>>>> _______________________________________________
> Modules-interest mailing list
> Mod...@li...
> https://lists.sourceforge.net/lists/listinfo/modules-interest
>
|
