|
From: Jonathan B. <jon...@gm...> - 2022-03-28 15:40:38
|
Hi Xavier,
I apologize for not sending a response sooner. You're completely right, I
missed that part. I implemented it successfully, thank you. And thanks
for helping and bearing with me!
-Jonathan
On Wed, Mar 23, 2022 at 12:34 PM Xavier Delaruelle <
xav...@gm...> wrote:
> Hi Jonathan,
>
> I see: you may I not seen the second part of my email from this morning
> :). Please find below the proposed new code for the module python function:
>
> import os, sys
> import re, subprocess
> from __future__ import print_function
> def module(*arguments):
> ns = {}
> out, err = subprocess.Popen(['/usr/bin/tclsh',
> '/usr/share/Modules/libexec/modulecmd.tcl', 'python'] + list(arguments),
> stdout=subprocess.PIPE, stderr=susprocess.PIPE).communicate()
> exec(out, ns)
> if '_mlstatus' in ns:
> _mlstatus = ns['_mlstatus']
> else:
> _mlstatus = True
> print(err.decode(), file=sys.stderr)
> return _mlstatus
>
> Regards,
> Xavier
>
> Le mer. 23 mars 2022 à 17:20, Jonathan Buck <jon...@gm...> a
> écrit :
>
>> Hi Xavier,
>>
>> Thanks for the additional information about _mlstatus. I tried to
>> implement the function as suggested, and I did not have the same success as
>> you. I may have done this completely wrong. I placed your function inside
>> of my own script, so that it's defined locally. I'm assuming that may be
>> the only difference between us, however I'm not sure why that would be a
>> problem. For what it's worth, my environment is:
>> bash version 4.4.23
>> python version 3.6.15
>> module version 4.7.1
>>
>> #!/usr/bin/python3
>>
>> import os
>> import sys
>> from io import StringIO
>> import re, subprocess
>> def module(*arguments):
>> ns = {}
>> exec(subprocess.Popen(['/usr/bin/tclsh',
>> '/usr/share/Modules/libexec/modulecmd.tcl', 'python'] + list(arguments),
>> stdout=subprocess.PIPE).communicate()[0], ns)
>> if '_mlstatus' in ns:
>> _mlstatus = ns['_mlstatus']
>> else:
>> _mlstatus = True
>> return _mlstatus
>>
>> # Setup variables to hold stdout/stderr
>> result_stdout = StringIO()
>> result_stderr = StringIO()
>> sys.stdout = result_stdout
>> sys.stderr = result_stderr
>>
>> # run
>> print("----- TRY RUNNING MODULE COMMAND -----")
>> module('--version')
>> print("----- DONE RUNNING MODULE COMMAND -----")
>> module_stdout = result_stdout.getvalue()
>> module_stderr = result_stderr.getvalue()
>>
>> # reset stdout/stderr
>> sys.stdout = sys.__stdout__
>> sys.stderr = sys.__stderr__
>>
>> # print out what was captured to the variables
>> print("===== TRY PRINTING STDOUT =====")
>> sys.stdout.write(module_stdout)
>> print("===== DONE PRINTING STDOUT =====")
>>
>> print("***** TRY PRINTING STDERR *****")
>> sys.stdout.write(module_stderr)
>> print("***** DONE PRINTING STDERR *****")
>>
>> The result of that script is:
>> ~> python3 module_test.py
>> Modules Release 4.7.1 (2021-04-06)
>> ===== TRY PRINTING STDOUT =====
>> ----- TRY RUNNING MODULE COMMAND -----
>> ----- DONE RUNNING MODULE COMMAND -----
>> ===== DONE PRINTING STDOUT =====
>> ***** TRY PRINTING STDERR *****
>> ***** DONE PRINTING STDERR *****
>>
>> On Wed, Mar 23, 2022 at 11:47 AM Xavier Delaruelle <
>> xav...@gm...> wrote:
>>
>>>
>>> Le mer. 23 mars 2022 à 16:31, Jonathan Buck <jon...@gm...> a
>>> écrit :
>>>
>>>> Hi Xavier,
>>>>
>>>> This is great! Thanks for showing me how to expose the underlying
>>>> function! One thing to note is that everything returned comes through
>>>> stderr. So, as you were saying, I wasn't able to trap anything because
>>>> everything was coming through stderr.
>>>>
>>>> In my testing, _mlstatus is returned by stdout. Are there other values
>>>> that might be returned by stdout other than "_mlstatus = False" ?
>>>>
>>>> Am I right in assuming '_mlstatus' is the return code of the tcl
>>>> execution?
>>>>
>>>
>>> _mlstatus is the variable that hold the return code of the evaluation.
>>> On stdout also goes all the environment changes implied by modulefile
>>> evaluation.
>>>
>>>
>>>> The only problem I still have is that I can't seem to capture that
>>>> stderr output to a variable. Currently, it goes to the console, no matter
>>>> what I do. I'm going to play around with it and see if this is because of
>>>> the exec() or the popen() statements. Ideally, I would like to capture the
>>>> output when the _mlstatus is False. For my use case, This feature would
>>>> allow me to compile a list of my 'bad' modules and their associated output.
>>>>
>>>> I've been playing with your initial code this morning, teasing it apart
>>>> and designing a return object containing the output and the return code.
>>>> This might not be the preferred methodology, but either way I'll report
>>>> back with what I find.
>>>>
>>>
>>> With the code I proposed this morning, you should be able to get the
>>> content from sys.stderr. On my side, I have run this proposed code in the
>>> python script you provided and I correctly got that stderr output between
>>> the TRY/DONE PRINTING STDERR lines.
>>>
>>> Regards,
>>> Xavier
>>>
>>>
>>>
>>>> Thanks again and with all respect,
>>>> Jonathan
>>>>
>>>>
>>>> On Wed, Mar 23, 2022 at 11:14 AM Xavier Delaruelle <
>>>> xav...@gm...> wrote:
>>>>
>>>>> Hi Jonathan,
>>>>>
>>>>> The exit status should be easy to grab from the module() python
>>>>> function. This function basically returns this exit status as a boolean. I
>>>>> paste below the current python code of the module() definition in python.
>>>>> It has not changed a lot since Modules v4.1.
>>>>>
>>>>> $ $MODULES_CMD python autoinit
>>>>> import os
>>>>> import re, subprocess
>>>>> def module(*arguments):
>>>>> ns = {}
>>>>> exec(subprocess.Popen(['/usr/bin/tclsh',
>>>>> '/usr/share/Modules/libexec/modulecmd.tcl', 'python'] + list(arguments),
>>>>> stdout=subprocess.PIPE).communicate()[0], ns)
>>>>> if '_mlstatus' in ns:
>>>>> _mlstatus = ns['_mlstatus']
>>>>> else:
>>>>> _mlstatus = True
>>>>> return _mlstatus
>>>>>
>>>>> With the code example you gave, the module function used is the one
>>>>> defined in a sh shell session (due to the shell=True parameter of
>>>>> subprocess.run). Usually the module shell function returns the output on
>>>>> stdout (stderr messages are redirected to stdout after the evaluation of
>>>>> the shell code generated on stdout).
>>>>>
>>>>> Regards,
>>>>> Xavier
>>>>>
>>>>> Le mer. 23 mars 2022 à 14:26, Jonathan Buck <jon...@gm...>
>>>>> a écrit :
>>>>>
>>>>>> Hi Paul,
>>>>>>
>>>>>> Thanks for the response! I took a look at this and spoke with a
>>>>>> colleague, but exit status is hard to grab from the module() method within
>>>>>> python. The only way we were able to pull this off was to ignore the
>>>>>> suggested python exec() statement found in the documentation and rather
>>>>>> create our own popen() statement. This allowed us to then check the status
>>>>>> of that process. What I don't like about this solution is that we're
>>>>>> depending on the originating shell to have properly sourced the module
>>>>>> environment, so that we can execute these 'os' commands. Here's a snippet
>>>>>> of the code my colleague came up with:
>>>>>>
>>>>>> for mod in modules_to_test: proc = subprocess.run(f"module use {mod_path};module load {mod}", stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True, universal_newlines=True)
>>>>>> print("", "########", f"Test: {mod}", "########", sep="\n") # Just FYI to see the attributes of the object # print(proc) print(f"stdout: {proc.stdout}") print(f"stderr: {proc.stderr}") print(f"return code: {proc.returncode}")
>>>>>>
>>>>>> We Found that all output goes to proc.stdout rather than any errors
>>>>>> going to proc.stderr, regardless of a return code being 1 or 0. The return
>>>>>> code is the winner here. So, this works but I'm calling it a dirty solution
>>>>>> because I think that once I automate this and try to execute remotely on
>>>>>> our clusters, we'll lose the properly sourced environment.
>>>>>>
>>>>>> On Tue, Mar 22, 2022 at 4:19 PM Paul Markfort <pau...@gm...>
>>>>>> wrote:
>>>>>>
>>>>>>> If you are just trying to, determine if there was an error thrown by
>>>>>>> the module command,
>>>>>>> Just test the exit status:
>>>>>>>
>>>>>>> EXIT STATUS
>>>>>>> The module command exits with 0 if its execution succeed.
>>>>>>> Otherwise 1
>>>>>>> is returned.
>>>>>>>
>>>>>>>
>>>>>>> Keep in mind that every call to the module command redirects
>>>>>>> sys.stdout so that the output is evaluated by python (so you probably can't
>>>>>>> capture that).
>>>>>>> And sys.stderr redirection by a module command might depend on what
>>>>>>> stderr is connected to.
>>>>>>>
>>>>>>> Last - stderr and stdout are buffered IO, so you may also have to do
>>>>>>> a flush before reading them.
>>>>>>>
>>>>>>>
>>>>>>> I should preface the below with, "I haven't coded in python":
>>>>>>>
>>>>>>> Maybe you need to test your code with no modules involved first.
>>>>>>> You might want to read this (seems like stringIO() may have some
>>>>>>> limitations).
>>>>>>>
>>>>>>> https://stackoverflow.com/questions/1218933/can-i-redirect-the-stdout-into-some-sort-of-string-buffer
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 2022-03-22 09:18 AM, Jonathan Buck wrote:
>>>>>>> > Hi Jacques,
>>>>>>> >
>>>>>>> > Thanks for the quick reply! I have successfully use the module()
>>>>>>> command
>>>>>>> > to setup environments without issue. What I'm hoping to do is
>>>>>>> capture any
>>>>>>> > stderr messages to a variable. I have several use cases for this,
>>>>>>> most
>>>>>>> > importantly I'm attempting to automate checks against our hundreds
>>>>>>> of
>>>>>>> > modules to determine if any of them are broken when loaded. I'm
>>>>>>> unable to
>>>>>>> > redirect stdout/stderr in a manner that I've used before which I
>>>>>>> believe is
>>>>>>> > due to the command being a subprocess. I'm not a python guru, but
>>>>>>> the
>>>>>>> > popen statement declares " stdout=subprocess.PIPE" which I thought
>>>>>>> was the
>>>>>>> > right way to get the stdout returned to the originating process.
>>>>>>> >
>>>>>>> > All that being said, here's my test script:
>>>>>>> >
>>>>>>> > import os
>>>>>>> > import sys
>>>>>>> > from io import StringIO
>>>>>>> >
>>>>>>> > # Setup variables to hold stdout/stderr
>>>>>>> > result_stdout = StringIO()
>>>>>>> > result_stderr = StringIO()
>>>>>>> > sys.stdout = result_stdout
>>>>>>> > sys.stderr = result_stderr
>>>>>>> >
>>>>>>> > # setup environment module
>>>>>>> > exec(open('/p/app/Modules/4.7.1/init/python.py').read())
>>>>>>> >
>>>>>>> > # run
>>>>>>> > print("----- TRY RUNNING MODULE COMMAND -----")
>>>>>>> > module('--version')
>>>>>>> > print("----- DONE RUNNING MODULE COMMAND -----")
>>>>>>> > module_stdout = result_stdout.getvalue()
>>>>>>> > module_stderr = result_stderr.getvalue()
>>>>>>> >
>>>>>>> > # reset stdout/stderr to defaults
>>>>>>> > sys.stdout = sys.__stdout__
>>>>>>> > sys.stderr = sys.__stderr__
>>>>>>> >
>>>>>>> > # print out what was captured to the variables
>>>>>>> > print("===== TRY PRINTING STDOUT =====")
>>>>>>> > sys.stdout.write(module_stdout)
>>>>>>> > print("===== DONE PRINTING STDOUT =====")
>>>>>>> >
>>>>>>> > print("***** TRY PRINTING STDERR *****")
>>>>>>> > sys.stderr.write(module_stderr)
>>>>>>> > print("***** DONE PRINTING STDERR *****")
>>>>>>> >
>>>>>>> >
>>>>>>> > As you can see, it's simple. I make sure to reset stdout/stderr to
>>>>>>> > defaults, however these are my results:
>>>>>>> >
>>>>>>> > ~> python3 module_test.py
>>>>>>> > Modules Release 4.7.1 (2021-04-06)
>>>>>>> > ===== TRY PRINTING JSON STDOUT =====
>>>>>>> > ----- TRY RUNNING MODULE COMMAND -----
>>>>>>> > ----- DONE RUNNING MODULE COMMAND -----
>>>>>>> > ===== DONE PRINTING JSON STDOUT =====
>>>>>>> > ***** TRY PRINTING JSON STDERR *****
>>>>>>> > ***** DONE PRINTING JSON STDERR *****
>>>>>>> > ~>
>>>>>>> >
>>>>>>> > This shows that my stdout redirect is capturing the 'print'
>>>>>>> statements
>>>>>>> > for running the module command. However the output for the module
>>>>>>> > command prints at the very beginning and doesn't show up. Any
>>>>>>> > thoughts? I found a fancy way to do the same thing in bash, and I
>>>>>>> > proved that in bash the same output is going to stdout.
>>>>>>> >
>>>>>>> >
>>>>>>> > Very Respectfully,
>>>>>>> >
>>>>>>> > Jonathan
>>>>>>> >
>>>>>>> >
>>>>>>> > On Tue, Mar 22, 2022 at 8:06 AM Contact <dv...@cr...> wrote:
>>>>>>> >
>>>>>>> >> Hello,
>>>>>>> >>
>>>>>>> >> In my case I used modules from python with the following logic:
>>>>>>> >>
>>>>>>> >> import os
>>>>>>> >> exec(open(/home/jrp/modules/init/python.py')).read())
>>>>>>> >> module('avail')
>>>>>>> >> module('load', 'gcc/7')
>>>>>>> >> module('list')
>>>>>>> >>
>>>>>>> >> From the above example,
>>>>>>> >>
>>>>>>> >> - exec call will allow me to use modules from python itself
>>>>>>> and define
>>>>>>> >> module function
>>>>>>> >> - Each module function invocation will be redirected to
>>>>>>> module to
>>>>>>> >> perform required action
>>>>>>> >>
>>>>>>> >> As reminder, the content of stdout is used by modules to setup all
>>>>>>> >> required environment functions, functions/alias (if supported by
>>>>>>> current
>>>>>>> >> shell), ...
>>>>>>> >>
>>>>>>> >> I hope it will help you.
>>>>>>> >>
>>>>>>> >> Jacques
>>>>>>> >>
>>>>>>> >>
>>>>>>> >> Le 21/03/2022 à 20:09, Jonathan Buck a écrit :
>>>>>>> >>
>>>>>>> >> Hi,
>>>>>>> >>
>>>>>>> >> I'm attempting to capture and act on the stdout and stderr of the
>>>>>>> module
>>>>>>> >> command. The python implementation appears to be a subprocess
>>>>>>> for which I
>>>>>>> >> wasn't able to redirect with io.StringIO() as I have with other
>>>>>>> output
>>>>>>> >> capture.
>>>>>>> >>
>>>>>>> >> I attempted to look through the code, which I admit I don't fully
>>>>>>> >> understand and appears to have a one-liner style of executing the
>>>>>>> >> subprocess. I'm writing code that integrates with a production
>>>>>>> system, so
>>>>>>> >> I'm not able to modify the python.py code.
>>>>>>> >>
>>>>>>> >> Does anyone know how else this might be accomplished? Or should
>>>>>>> I make
>>>>>>> >> this a feature request?
>>>>>>> >>
>>>>>>> >> Thanks for your help,
>>>>>>> >> Jonathan
>>>>>>>
>>>>>> _______________________________________________
>>> Modules-interest mailing list
>>> Mod...@li...
>>> https://lists.sourceforge.net/lists/listinfo/modules-interest
>>>
>> _______________________________________________
>> Modules-interest mailing list
>> Mod...@li...
>> https://lists.sourceforge.net/lists/listinfo/modules-interest
>>
> _______________________________________________
> Modules-interest mailing list
> Mod...@li...
> https://lists.sourceforge.net/lists/listinfo/modules-interest
>
|
