|
From: Jonathan B. <jon...@gm...> - 2022-03-23 15:30:51
|
Hi Xavier,
This is great! Thanks for showing me how to expose the underlying
function! One thing to note is that everything returned comes through
stderr. So, as you were saying, I wasn't able to trap anything because
everything was coming through stderr.
In my testing, _mlstatus is returned by stdout. Are there other values
that might be returned by stdout other than "_mlstatus = False" ?
Am I right in assuming '_mlstatus' is the return code of the tcl execution?
The only problem I still have is that I can't seem to capture that stderr
output to a variable. Currently, it goes to the console, no matter what I
do. I'm going to play around with it and see if this is because of the
exec() or the popen() statements. Ideally, I would like to capture the
output when the _mlstatus is False. For my use case, This feature would
allow me to compile a list of my 'bad' modules and their associated output.
I've been playing with your initial code this morning, teasing it apart and
designing a return object containing the output and the return code. This
might not be the preferred methodology, but either way I'll report back
with what I find.
Thanks again and with all respect,
Jonathan
On Wed, Mar 23, 2022 at 11:14 AM Xavier Delaruelle <
xav...@gm...> wrote:
> Hi Jonathan,
>
> The exit status should be easy to grab from the module() python function.
> This function basically returns this exit status as a boolean. I paste
> below the current python code of the module() definition in python. It has
> not changed a lot since Modules v4.1.
>
> $ $MODULES_CMD python autoinit
> import os
> import re, subprocess
> def module(*arguments):
> ns = {}
> exec(subprocess.Popen(['/usr/bin/tclsh',
> '/usr/share/Modules/libexec/modulecmd.tcl', 'python'] + list(arguments),
> stdout=subprocess.PIPE).communicate()[0], ns)
> if '_mlstatus' in ns:
> _mlstatus = ns['_mlstatus']
> else:
> _mlstatus = True
> return _mlstatus
>
> With the code example you gave, the module function used is the one
> defined in a sh shell session (due to the shell=True parameter of
> subprocess.run). Usually the module shell function returns the output on
> stdout (stderr messages are redirected to stdout after the evaluation of
> the shell code generated on stdout).
>
> Regards,
> Xavier
>
> Le mer. 23 mars 2022 à 14:26, Jonathan Buck <jon...@gm...> a
> écrit :
>
>> Hi Paul,
>>
>> Thanks for the response! I took a look at this and spoke with a
>> colleague, but exit status is hard to grab from the module() method within
>> python. The only way we were able to pull this off was to ignore the
>> suggested python exec() statement found in the documentation and rather
>> create our own popen() statement. This allowed us to then check the status
>> of that process. What I don't like about this solution is that we're
>> depending on the originating shell to have properly sourced the module
>> environment, so that we can execute these 'os' commands. Here's a snippet
>> of the code my colleague came up with:
>>
>> for mod in modules_to_test: proc = subprocess.run(f"module use {mod_path};module load {mod}", stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True, universal_newlines=True)
>> print("", "########", f"Test: {mod}", "########", sep="\n") # Just FYI to see the attributes of the object # print(proc) print(f"stdout: {proc.stdout}") print(f"stderr: {proc.stderr}") print(f"return code: {proc.returncode}")
>>
>> We Found that all output goes to proc.stdout rather than any errors going
>> to proc.stderr, regardless of a return code being 1 or 0. The return code
>> is the winner here. So, this works but I'm calling it a dirty solution
>> because I think that once I automate this and try to execute remotely on
>> our clusters, we'll lose the properly sourced environment.
>>
>> On Tue, Mar 22, 2022 at 4:19 PM Paul Markfort <pau...@gm...>
>> wrote:
>>
>>> If you are just trying to, determine if there was an error thrown by the
>>> module command,
>>> Just test the exit status:
>>>
>>> EXIT STATUS
>>> The module command exits with 0 if its execution succeed.
>>> Otherwise 1
>>> is returned.
>>>
>>>
>>> Keep in mind that every call to the module command redirects sys.stdout
>>> so that the output is evaluated by python (so you probably can't capture
>>> that).
>>> And sys.stderr redirection by a module command might depend on what
>>> stderr is connected to.
>>>
>>> Last - stderr and stdout are buffered IO, so you may also have to do a
>>> flush before reading them.
>>>
>>>
>>> I should preface the below with, "I haven't coded in python":
>>>
>>> Maybe you need to test your code with no modules involved first.
>>> You might want to read this (seems like stringIO() may have some
>>> limitations).
>>>
>>> https://stackoverflow.com/questions/1218933/can-i-redirect-the-stdout-into-some-sort-of-string-buffer
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On 2022-03-22 09:18 AM, Jonathan Buck wrote:
>>> > Hi Jacques,
>>> >
>>> > Thanks for the quick reply! I have successfully use the module()
>>> command
>>> > to setup environments without issue. What I'm hoping to do is capture
>>> any
>>> > stderr messages to a variable. I have several use cases for this, most
>>> > importantly I'm attempting to automate checks against our hundreds of
>>> > modules to determine if any of them are broken when loaded. I'm
>>> unable to
>>> > redirect stdout/stderr in a manner that I've used before which I
>>> believe is
>>> > due to the command being a subprocess. I'm not a python guru, but the
>>> > popen statement declares " stdout=subprocess.PIPE" which I thought was
>>> the
>>> > right way to get the stdout returned to the originating process.
>>> >
>>> > All that being said, here's my test script:
>>> >
>>> > import os
>>> > import sys
>>> > from io import StringIO
>>> >
>>> > # Setup variables to hold stdout/stderr
>>> > result_stdout = StringIO()
>>> > result_stderr = StringIO()
>>> > sys.stdout = result_stdout
>>> > sys.stderr = result_stderr
>>> >
>>> > # setup environment module
>>> > exec(open('/p/app/Modules/4.7.1/init/python.py').read())
>>> >
>>> > # run
>>> > print("----- TRY RUNNING MODULE COMMAND -----")
>>> > module('--version')
>>> > print("----- DONE RUNNING MODULE COMMAND -----")
>>> > module_stdout = result_stdout.getvalue()
>>> > module_stderr = result_stderr.getvalue()
>>> >
>>> > # reset stdout/stderr to defaults
>>> > sys.stdout = sys.__stdout__
>>> > sys.stderr = sys.__stderr__
>>> >
>>> > # print out what was captured to the variables
>>> > print("===== TRY PRINTING STDOUT =====")
>>> > sys.stdout.write(module_stdout)
>>> > print("===== DONE PRINTING STDOUT =====")
>>> >
>>> > print("***** TRY PRINTING STDERR *****")
>>> > sys.stderr.write(module_stderr)
>>> > print("***** DONE PRINTING STDERR *****")
>>> >
>>> >
>>> > As you can see, it's simple. I make sure to reset stdout/stderr to
>>> > defaults, however these are my results:
>>> >
>>> > ~> python3 module_test.py
>>> > Modules Release 4.7.1 (2021-04-06)
>>> > ===== TRY PRINTING JSON STDOUT =====
>>> > ----- TRY RUNNING MODULE COMMAND -----
>>> > ----- DONE RUNNING MODULE COMMAND -----
>>> > ===== DONE PRINTING JSON STDOUT =====
>>> > ***** TRY PRINTING JSON STDERR *****
>>> > ***** DONE PRINTING JSON STDERR *****
>>> > ~>
>>> >
>>> > This shows that my stdout redirect is capturing the 'print' statements
>>> > for running the module command. However the output for the module
>>> > command prints at the very beginning and doesn't show up. Any
>>> > thoughts? I found a fancy way to do the same thing in bash, and I
>>> > proved that in bash the same output is going to stdout.
>>> >
>>> >
>>> > Very Respectfully,
>>> >
>>> > Jonathan
>>> >
>>> >
>>> > On Tue, Mar 22, 2022 at 8:06 AM Contact <dv...@cr...> wrote:
>>> >
>>> >> Hello,
>>> >>
>>> >> In my case I used modules from python with the following logic:
>>> >>
>>> >> import os
>>> >> exec(open(/home/jrp/modules/init/python.py')).read())
>>> >> module('avail')
>>> >> module('load', 'gcc/7')
>>> >> module('list')
>>> >>
>>> >> From the above example,
>>> >>
>>> >> - exec call will allow me to use modules from python itself and
>>> define
>>> >> module function
>>> >> - Each module function invocation will be redirected to module to
>>> >> perform required action
>>> >>
>>> >> As reminder, the content of stdout is used by modules to setup all
>>> >> required environment functions, functions/alias (if supported by
>>> current
>>> >> shell), ...
>>> >>
>>> >> I hope it will help you.
>>> >>
>>> >> Jacques
>>> >>
>>> >>
>>> >> Le 21/03/2022 à 20:09, Jonathan Buck a écrit :
>>> >>
>>> >> Hi,
>>> >>
>>> >> I'm attempting to capture and act on the stdout and stderr of the
>>> module
>>> >> command. The python implementation appears to be a subprocess for
>>> which I
>>> >> wasn't able to redirect with io.StringIO() as I have with other output
>>> >> capture.
>>> >>
>>> >> I attempted to look through the code, which I admit I don't fully
>>> >> understand and appears to have a one-liner style of executing the
>>> >> subprocess. I'm writing code that integrates with a production
>>> system, so
>>> >> I'm not able to modify the python.py code.
>>> >>
>>> >> Does anyone know how else this might be accomplished? Or should I
>>> make
>>> >> this a feature request?
>>> >>
>>> >> Thanks for your help,
>>> >> Jonathan
>>> >>
>>> >>
>>> >> _______________________________________________
>>> >> Modules-interest mailing listModules-interest
>>> @lists.sourceforge.nethttps://
>>> lists.sourceforge.net/lists/listinfo/modules-interest
>>> >>
>>> >>
>>> >
>>> >
>>> >
>>> > _______________________________________________
>>> > Modules-interest mailing list
>>> > Mod...@li...
>>> > https://lists.sourceforge.net/lists/listinfo/modules-interest
>>> >
>>>
>>> --
>>> --------------------------------------------------------
>>> The views and opinions expressed above are strictly
>>> those of the author(s). The content of this message has
>>> not been reviewed nor approved by any entity whatsoever.
>>> --------------------------------------------------------
>>> Paul FM Info: http://paulfm.com/~paulfm/
>>> --------------------------------------------------------
>>>
>>>
>>> _______________________________________________
>>> Modules-interest mailing list
>>> Mod...@li...
>>> https://lists.sourceforge.net/lists/listinfo/modules-interest
>>>
>> _______________________________________________
>> Modules-interest mailing list
>> Mod...@li...
>> https://lists.sourceforge.net/lists/listinfo/modules-interest
>>
> _______________________________________________
> Modules-interest mailing list
> Mod...@li...
> https://lists.sourceforge.net/lists/listinfo/modules-interest
>
|
