modules-interest — Discussion forum for modules usage.

Re: [Modules] opening python in the cron

[Xavier D. <xav...@gm...>]

Hi Charles,

Could you provide the script you use to reproduce this issue on my side
(you could send it to me directly if you do not want to post this script on
the mailing-list).

Regards,
Xavier

Le jeu. 24 mars 2022 à 14:49, Charles Ross - NOAA Federal via
Modules-interest <mod...@li...> a écrit :

> Hi all,
>
> I hope this isn't too basic of a question.
>
> I have a shell script that opens the python module and then executes a
> python script. Pretty basic and simple, works well.
>
> The same script will not work on the cron.  It is not able to successfully
> pass the python version opened through modules.
>
> Any help on what to do would be much appreciated,
>
> Charles Ross
>
>
> _______________________________________________
> Modules-interest mailing list
> Mod...@li...
> https://lists.sourceforge.net/lists/listinfo/modules-interest
>

[Modules] opening python in the cron

[Charles R. - N. F. <cha...@no...>]

Hi all,

I hope this isn't too basic of a question.

I have a shell script that opens the python module and then executes a
python script. Pretty basic and simple, works well.

The same script will not work on the cron.  It is not able to successfully
pass the python version opened through modules.

Any help on what to do would be much appreciated,

Charles Ross

Re: [Modules] Does a linter for module files exist?

[Erik P. <eri...@si...>]

Thanks for trying it out! I’ll see if I can get something simple working with the .syntax files where I just support a couple of the additional procedures brought by the modulefile interpreter.

> On Mar 22, 2022, at 10:43 PM, Xavier Delaruelle <xav...@gm...> wrote:
> 
> The "source syntaxbuild.tcl" + "buildFile mysyntaxdb.tcl" approach described in the Nagelfar documentation does not seem to work when called from a modulefile evaluation context. I have seen that there are also ".syntax" files : maybe that is a simpler approach to describe in such files the additionnal procedures and variables brought by the modulefile interpreter. But I do not know if global variables are supported in these .syntax file.
> 
> Regards,
> Xavier
> 
> Le lun. 21 mars 2022 à 21:35, Erik Parawell <eri...@si... <mailto:eri...@si...>> a écrit :
> Thanks Xavier for the detailed answer!
> 
> I’ll check out that formatting script and give it a test.
> 
> In regards to Nagelfar, I’m looking into the syntaxbuild.tcl and it looks like it is partially hardcoded and then tries to extract things from the interpreter like subcommands. http://nagelfar.sourceforge.net/syntaxdatabases.html#creating-syntax-databases <http://nagelfar.sourceforge.net/syntaxdatabases.html#creating-syntax-databases>
> 
> Do you know how to can run tcl scripts from the modulefile interpreter?
> 
>> On Mar 19, 2022, at 5:33 AM, Xavier Delaruelle <xav...@gm... <mailto:xav...@gm...>> wrote:
>> 
>> 
>> Le ven. 18 mars 2022 à 22:01, Erik Parawell <eri...@si... <mailto:eri...@si...>> a écrit :
>> Hi,
>> 
>> I’m managing a lot of module files and I was wondering if there is a linter for modules files or maybe some formatter (espeically for indentation). I have seen that nagelfar132 is a good linter for TCL code and that you can create a new syntax database using a provided scrip, but I can’t figure out if I can even run that script with the module interpreter.
>> 
>> Suggestions are welcome!
>> 
>> Regards,
>> Erik Parawell
>> 
>> Hi Erik,
>> 
>> As far as I know there is no linter existing for modulefiles. Best thing available at the moment is to use "module test" against the modulefiles to see if there are syntax errors.
>> 
>> Using Nagelfar for linting seems quite interesting. Need to figure out how a new syntax db could be generated with all Tcl commands and variables added by the modulefile interpreter.
>> 
>> Regarding code formatter, it seems there are not a lot of solution available out there. I have found (but not tested) this script proposed by the Tcl community:
>> https://wiki.tcl-lang.org/page/Reformatting+Tcl+code+indentation <https://wiki.tcl-lang.org/page/Reformatting+Tcl+code+indentation>
>> 
>> Regards,
>> Xavier
>> _______________________________________________
>> Modules-interest mailing list
>> Mod...@li... <mailto:Mod...@li...>
>> https://lists.sourceforge.net/lists/listinfo/modules-interest <https://lists.sourceforge.net/lists/listinfo/modules-interest>
> 
> _______________________________________________
> Modules-interest mailing list
> Mod...@li... <mailto:Mod...@li...>
> https://lists.sourceforge.net/lists/listinfo/modules-interest <https://lists.sourceforge.net/lists/listinfo/modules-interest>
> _______________________________________________
> Modules-interest mailing list
> Mod...@li...
> https://lists.sourceforge.net/lists/listinfo/modules-interest

Re: [Modules] Capture python stdout and stderr

[Xavier D. <xav...@gm...>]

Hi Jonathan,

I see: you may I not seen the second part of my email from this morning :).
Please find below the proposed new code for the module python function:

import os, sys
import re, subprocess
from __future__ import print_function
def module(*arguments):
   ns = {}
   out, err = subprocess.Popen(['/usr/bin/tclsh',
'/usr/share/Modules/libexec/modulecmd.tcl', 'python'] + list(arguments),
stdout=subprocess.PIPE, stderr=susprocess.PIPE).communicate()
   exec(out, ns)
   if '_mlstatus' in ns:
      _mlstatus = ns['_mlstatus']
   else:
      _mlstatus = True
   print(err.decode(), file=sys.stderr)
   return _mlstatus

Regards,
Xavier

Le mer. 23 mars 2022 à 17:20, Jonathan Buck <jon...@gm...> a
écrit :

> Hi Xavier,
>
> Thanks for the additional information about _mlstatus.  I tried to
> implement the function as suggested, and I did not have the same success as
> you.  I may have done this completely wrong.  I placed your function inside
> of my own script, so that it's defined locally.  I'm assuming that may be
> the only difference between us, however I'm not sure why that would be a
> problem. For what it's worth, my environment is:
> bash version 4.4.23
> python version 3.6.15
> module version 4.7.1
>
> #!/usr/bin/python3
>
> import os
> import sys
> from io import StringIO
> import re, subprocess
> def module(*arguments):
>     ns = {}
>     exec(subprocess.Popen(['/usr/bin/tclsh',
> '/usr/share/Modules/libexec/modulecmd.tcl', 'python'] + list(arguments),
> stdout=subprocess.PIPE).communicate()[0], ns)
>     if '_mlstatus' in ns:
>         _mlstatus = ns['_mlstatus']
>     else:
>         _mlstatus = True
>     return _mlstatus
>
> # Setup variables to hold stdout/stderr
> result_stdout = StringIO()
> result_stderr = StringIO()
> sys.stdout = result_stdout
> sys.stderr = result_stderr
>
> # run
> print("----- TRY RUNNING MODULE COMMAND -----")
> module('--version')
> print("----- DONE RUNNING MODULE COMMAND -----")
> module_stdout = result_stdout.getvalue()
> module_stderr = result_stderr.getvalue()
>
> # reset stdout/stderr
> sys.stdout = sys.__stdout__
> sys.stderr = sys.__stderr__
>
> # print out what was captured to the variables
> print("===== TRY PRINTING STDOUT =====")
> sys.stdout.write(module_stdout)
> print("===== DONE PRINTING STDOUT =====")
>
> print("***** TRY PRINTING STDERR *****")
> sys.stdout.write(module_stderr)
> print("***** DONE PRINTING STDERR *****")
>
> The result of that script is:
> ~> python3 module_test.py
> Modules Release 4.7.1 (2021-04-06)
> ===== TRY PRINTING STDOUT =====
> ----- TRY RUNNING MODULE COMMAND -----
> ----- DONE RUNNING MODULE COMMAND -----
> ===== DONE PRINTING STDOUT =====
> ***** TRY PRINTING STDERR *****
> ***** DONE PRINTING STDERR *****
>
> On Wed, Mar 23, 2022 at 11:47 AM Xavier Delaruelle <
> xav...@gm...> wrote:
>
>>
>> Le mer. 23 mars 2022 à 16:31, Jonathan Buck <jon...@gm...> a
>> écrit :
>>
>>>   Hi Xavier,
>>>
>>> This is great!  Thanks for showing me how to expose the underlying
>>> function!  One thing to note is that everything returned comes through
>>> stderr.  So, as you were saying, I wasn't able to trap anything because
>>> everything was coming through stderr.
>>>
>>> In my testing, _mlstatus is returned by stdout.  Are there other values
>>> that might be returned by stdout other than "_mlstatus = False" ?
>>>
>>> Am I right in assuming '_mlstatus' is the return code of the tcl
>>> execution?
>>>
>>
>> _mlstatus is the variable that hold the return code of the evaluation. On
>> stdout also goes all the environment changes implied by modulefile
>> evaluation.
>>
>>
>>> The only problem I still have is that I can't seem to capture that
>>> stderr output to a variable.  Currently, it goes to the console, no matter
>>> what I do.  I'm going to play around with it and see if this is because of
>>> the exec() or the popen() statements.  Ideally, I would like to capture the
>>> output when the _mlstatus is False.  For my use case, This feature would
>>> allow me to compile a list of my 'bad' modules and their associated output.
>>>
>>> I've been playing with your initial code this morning, teasing it apart
>>> and designing a return object containing the output and the return code.
>>> This might not be the preferred methodology, but either way I'll report
>>> back with what I find.
>>>
>>
>> With the code I proposed this morning, you should be able to get the
>> content from sys.stderr. On my side, I have run this proposed code in the
>> python script you provided and I correctly got that stderr output between
>> the TRY/DONE PRINTING STDERR lines.
>>
>> Regards,
>> Xavier
>>
>>
>>
>>> Thanks again and with all respect,
>>> Jonathan
>>>
>>>
>>> On Wed, Mar 23, 2022 at 11:14 AM Xavier Delaruelle <
>>> xav...@gm...> wrote:
>>>
>>>> Hi Jonathan,
>>>>
>>>> The exit status should be easy to grab from the module() python
>>>> function. This function basically returns this exit status as a boolean. I
>>>> paste below the current python code of the module() definition in python.
>>>> It has not changed a lot since Modules v4.1.
>>>>
>>>> $ $MODULES_CMD python autoinit
>>>> import os
>>>> import re, subprocess
>>>> def module(*arguments):
>>>>    ns = {}
>>>>    exec(subprocess.Popen(['/usr/bin/tclsh',
>>>> '/usr/share/Modules/libexec/modulecmd.tcl', 'python'] + list(arguments),
>>>> stdout=subprocess.PIPE).communicate()[0], ns)
>>>>    if '_mlstatus' in ns:
>>>>       _mlstatus = ns['_mlstatus']
>>>>    else:
>>>>       _mlstatus = True
>>>>    return _mlstatus
>>>>
>>>> With the code example you gave, the module function used is the one
>>>> defined in a sh shell session (due to the shell=True parameter of
>>>> subprocess.run). Usually the module shell function returns the output on
>>>> stdout (stderr messages are redirected to stdout after the evaluation of
>>>> the shell code generated on stdout).
>>>>
>>>> Regards,
>>>> Xavier
>>>>
>>>> Le mer. 23 mars 2022 à 14:26, Jonathan Buck <jon...@gm...>
>>>> a écrit :
>>>>
>>>>> Hi Paul,
>>>>>
>>>>> Thanks for the response!  I took a look at this and spoke with a
>>>>> colleague, but exit status is hard to grab from the module() method within
>>>>> python.  The only way we were able to pull this off was to ignore the
>>>>> suggested python exec() statement found in the documentation and rather
>>>>> create our own popen() statement.  This allowed us to then check the status
>>>>> of that process.  What I don't like about this solution is that we're
>>>>> depending on the originating shell to have properly sourced the module
>>>>> environment, so that we can execute these 'os' commands.  Here's a snippet
>>>>> of the code my colleague came up with:
>>>>>
>>>>> for mod in modules_to_test:    proc = subprocess.run(f"module use {mod_path};module load {mod}",            stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True,            universal_newlines=True)
>>>>>     print("", "########", f"Test: {mod}", "########", sep="\n")    # Just FYI to see the attributes of the object    # print(proc)    print(f"stdout: {proc.stdout}")    print(f"stderr: {proc.stderr}")    print(f"return code: {proc.returncode}")
>>>>>
>>>>> We Found that all output goes to proc.stdout rather than any errors
>>>>> going to proc.stderr, regardless of a return code being 1 or 0.  The return
>>>>> code is the winner here. So, this works but I'm calling it a dirty solution
>>>>> because I think that once I automate this and try to execute remotely on
>>>>> our clusters, we'll lose the properly sourced environment.
>>>>>
>>>>> On Tue, Mar 22, 2022 at 4:19 PM Paul Markfort <pau...@gm...>
>>>>> wrote:
>>>>>
>>>>>> If you are just trying to, determine if there was an error thrown by
>>>>>> the module command,
>>>>>> Just test the exit status:
>>>>>>
>>>>>> EXIT STATUS
>>>>>>         The module command exits with 0 if its execution succeed.
>>>>>> Otherwise  1
>>>>>>         is returned.
>>>>>>
>>>>>>
>>>>>> Keep in mind that every call to the module command redirects
>>>>>> sys.stdout so that the output is evaluated by python (so you probably can't
>>>>>> capture that).
>>>>>> And sys.stderr redirection by a module command might depend on what
>>>>>> stderr is connected to.
>>>>>>
>>>>>> Last - stderr and stdout are buffered IO, so you may also have to do
>>>>>> a flush before reading them.
>>>>>>
>>>>>>
>>>>>> I should preface the below with, "I haven't coded in python":
>>>>>>
>>>>>> Maybe you need to test your code with no modules involved first.
>>>>>> You might want to read this (seems like stringIO() may have some
>>>>>> limitations).
>>>>>>
>>>>>> https://stackoverflow.com/questions/1218933/can-i-redirect-the-stdout-into-some-sort-of-string-buffer
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 2022-03-22 09:18 AM, Jonathan Buck wrote:
>>>>>> > Hi Jacques,
>>>>>> >
>>>>>> > Thanks for the quick reply!  I have successfully use the module()
>>>>>> command
>>>>>> > to setup environments without issue.  What I'm hoping to do is
>>>>>> capture any
>>>>>> > stderr messages to a variable.  I have several use cases for this,
>>>>>> most
>>>>>> > importantly I'm attempting to automate checks against our hundreds
>>>>>> of
>>>>>> > modules to determine if any of them are broken when loaded.  I'm
>>>>>> unable to
>>>>>> > redirect stdout/stderr in a manner that I've used before which I
>>>>>> believe is
>>>>>> > due to the command being a subprocess.  I'm not a python guru, but
>>>>>> the
>>>>>> > popen statement declares " stdout=subprocess.PIPE" which I thought
>>>>>> was the
>>>>>> > right way to get the stdout returned to the originating process.
>>>>>> >
>>>>>> > All that being said, here's my test script:
>>>>>> >
>>>>>> > import os
>>>>>> > import sys
>>>>>> > from io import StringIO
>>>>>> >
>>>>>> > # Setup variables to hold stdout/stderr
>>>>>> > result_stdout = StringIO()
>>>>>> > result_stderr = StringIO()
>>>>>> > sys.stdout = result_stdout
>>>>>> > sys.stderr = result_stderr
>>>>>> >
>>>>>> > # setup environment module
>>>>>> > exec(open('/p/app/Modules/4.7.1/init/python.py').read())
>>>>>> >
>>>>>> > # run
>>>>>> > print("----- TRY RUNNING MODULE COMMAND -----")
>>>>>> > module('--version')
>>>>>> > print("----- DONE RUNNING MODULE COMMAND -----")
>>>>>> > module_stdout = result_stdout.getvalue()
>>>>>> > module_stderr = result_stderr.getvalue()
>>>>>> >
>>>>>> > # reset stdout/stderr to defaults
>>>>>> > sys.stdout = sys.__stdout__
>>>>>> > sys.stderr = sys.__stderr__
>>>>>> >
>>>>>> > # print out what was captured to the variables
>>>>>> > print("===== TRY PRINTING STDOUT =====")
>>>>>> > sys.stdout.write(module_stdout)
>>>>>> > print("===== DONE PRINTING STDOUT =====")
>>>>>> >
>>>>>> > print("***** TRY PRINTING STDERR *****")
>>>>>> > sys.stderr.write(module_stderr)
>>>>>> > print("***** DONE PRINTING STDERR *****")
>>>>>> >
>>>>>> >
>>>>>> > As you can see, it's simple.  I make sure to reset stdout/stderr to
>>>>>> > defaults, however these are my results:
>>>>>> >
>>>>>> > ~> python3 module_test.py
>>>>>> > Modules Release 4.7.1 (2021-04-06)
>>>>>> > ===== TRY PRINTING JSON STDOUT =====
>>>>>> > ----- TRY RUNNING MODULE COMMAND -----
>>>>>> > ----- DONE RUNNING MODULE COMMAND -----
>>>>>> > ===== DONE PRINTING JSON STDOUT =====
>>>>>> > ***** TRY PRINTING JSON STDERR *****
>>>>>> > ***** DONE PRINTING JSON STDERR *****
>>>>>> > ~>
>>>>>> >
>>>>>> > This shows that my stdout redirect is capturing the 'print'
>>>>>> statements
>>>>>> > for running the module command.  However the output for the module
>>>>>> > command prints at the very beginning and doesn't show up.  Any
>>>>>> > thoughts?  I found a fancy way to do the same thing in bash, and I
>>>>>> > proved that in bash the same output is going to stdout.
>>>>>> >
>>>>>> >
>>>>>> > Very Respectfully,
>>>>>> >
>>>>>> > Jonathan
>>>>>> >
>>>>>> >
>>>>>> > On Tue, Mar 22, 2022 at 8:06 AM Contact <dv...@cr...> wrote:
>>>>>> >
>>>>>> >> Hello,
>>>>>> >>
>>>>>> >> In my case I used modules from python with the following logic:
>>>>>> >>
>>>>>> >> import os
>>>>>> >> exec(open(/home/jrp/modules/init/python.py')).read())
>>>>>> >> module('avail')
>>>>>> >> module('load', 'gcc/7')
>>>>>> >> module('list')
>>>>>> >>
>>>>>> >>  From the above example,
>>>>>> >>
>>>>>> >>     - exec call will allow me to use modules from python itself
>>>>>> and define
>>>>>> >>     module function
>>>>>> >>     - Each module function invocation will be redirected to module
>>>>>> to
>>>>>> >>     perform required action
>>>>>> >>
>>>>>> >> As reminder, the content of stdout is used by modules to setup all
>>>>>> >> required environment functions, functions/alias (if supported by
>>>>>> current
>>>>>> >> shell), ...
>>>>>> >>
>>>>>> >> I hope it will help you.
>>>>>> >>
>>>>>> >> Jacques
>>>>>> >>
>>>>>> >>
>>>>>> >> Le 21/03/2022 à 20:09, Jonathan Buck a écrit :
>>>>>> >>
>>>>>> >> Hi,
>>>>>> >>
>>>>>> >> I'm attempting to capture and act on the stdout and stderr of the
>>>>>> module
>>>>>> >> command.  The python implementation appears to be a subprocess for
>>>>>> which I
>>>>>> >> wasn't able to redirect with io.StringIO() as I have with other
>>>>>> output
>>>>>> >> capture.
>>>>>> >>
>>>>>> >> I attempted to look through the code, which I admit I don't fully
>>>>>> >> understand and appears to have a one-liner style of executing the
>>>>>> >> subprocess.  I'm writing code that integrates with a production
>>>>>> system, so
>>>>>> >> I'm not able to modify the python.py code.
>>>>>> >>
>>>>>> >> Does anyone know how else this might be accomplished?  Or should I
>>>>>> make
>>>>>> >> this a feature request?
>>>>>> >>
>>>>>> >> Thanks for your help,
>>>>>> >> Jonathan
>>>>>>
>>>>> _______________________________________________
>> Modules-interest mailing list
>> Mod...@li...
>> https://lists.sourceforge.net/lists/listinfo/modules-interest
>>
> _______________________________________________
> Modules-interest mailing list
> Mod...@li...
> https://lists.sourceforge.net/lists/listinfo/modules-interest
>

Re: [Modules] Capture python stdout and stderr

[Jonathan B. <jon...@gm...>]

Hi Xavier,

Thanks for the additional information about _mlstatus.  I tried to
implement the function as suggested, and I did not have the same success as
you.  I may have done this completely wrong.  I placed your function inside
of my own script, so that it's defined locally.  I'm assuming that may be
the only difference between us, however I'm not sure why that would be a
problem. For what it's worth, my environment is:
bash version 4.4.23
python version 3.6.15
module version 4.7.1

#!/usr/bin/python3

import os
import sys
from io import StringIO
import re, subprocess
def module(*arguments):
    ns = {}
    exec(subprocess.Popen(['/usr/bin/tclsh',
'/usr/share/Modules/libexec/modulecmd.tcl', 'python'] + list(arguments),
stdout=subprocess.PIPE).communicate()[0], ns)
    if '_mlstatus' in ns:
        _mlstatus = ns['_mlstatus']
    else:
        _mlstatus = True
    return _mlstatus

# Setup variables to hold stdout/stderr
result_stdout = StringIO()
result_stderr = StringIO()
sys.stdout = result_stdout
sys.stderr = result_stderr

# run
print("----- TRY RUNNING MODULE COMMAND -----")
module('--version')
print("----- DONE RUNNING MODULE COMMAND -----")
module_stdout = result_stdout.getvalue()
module_stderr = result_stderr.getvalue()

# reset stdout/stderr
sys.stdout = sys.__stdout__
sys.stderr = sys.__stderr__

# print out what was captured to the variables
print("===== TRY PRINTING STDOUT =====")
sys.stdout.write(module_stdout)
print("===== DONE PRINTING STDOUT =====")

print("***** TRY PRINTING STDERR *****")
sys.stdout.write(module_stderr)
print("***** DONE PRINTING STDERR *****")

The result of that script is:
~> python3 module_test.py
Modules Release 4.7.1 (2021-04-06)
===== TRY PRINTING STDOUT =====
----- TRY RUNNING MODULE COMMAND -----
----- DONE RUNNING MODULE COMMAND -----
===== DONE PRINTING STDOUT =====
***** TRY PRINTING STDERR *****
***** DONE PRINTING STDERR *****

On Wed, Mar 23, 2022 at 11:47 AM Xavier Delaruelle <
xav...@gm...> wrote:

>
> Le mer. 23 mars 2022 à 16:31, Jonathan Buck <jon...@gm...> a
> écrit :
>
>>   Hi Xavier,
>>
>> This is great!  Thanks for showing me how to expose the underlying
>> function!  One thing to note is that everything returned comes through
>> stderr.  So, as you were saying, I wasn't able to trap anything because
>> everything was coming through stderr.
>>
>> In my testing, _mlstatus is returned by stdout.  Are there other values
>> that might be returned by stdout other than "_mlstatus = False" ?
>>
>> Am I right in assuming '_mlstatus' is the return code of the tcl
>> execution?
>>
>
> _mlstatus is the variable that hold the return code of the evaluation. On
> stdout also goes all the environment changes implied by modulefile
> evaluation.
>
>
>> The only problem I still have is that I can't seem to capture that stderr
>> output to a variable.  Currently, it goes to the console, no matter what I
>> do.  I'm going to play around with it and see if this is because of the
>> exec() or the popen() statements.  Ideally, I would like to capture the
>> output when the _mlstatus is False.  For my use case, This feature would
>> allow me to compile a list of my 'bad' modules and their associated output.
>>
>> I've been playing with your initial code this morning, teasing it apart
>> and designing a return object containing the output and the return code.
>> This might not be the preferred methodology, but either way I'll report
>> back with what I find.
>>
>
> With the code I proposed this morning, you should be able to get the
> content from sys.stderr. On my side, I have run this proposed code in the
> python script you provided and I correctly got that stderr output between
> the TRY/DONE PRINTING STDERR lines.
>
> Regards,
> Xavier
>
>
>
>> Thanks again and with all respect,
>> Jonathan
>>
>>
>> On Wed, Mar 23, 2022 at 11:14 AM Xavier Delaruelle <
>> xav...@gm...> wrote:
>>
>>> Hi Jonathan,
>>>
>>> The exit status should be easy to grab from the module() python
>>> function. This function basically returns this exit status as a boolean. I
>>> paste below the current python code of the module() definition in python.
>>> It has not changed a lot since Modules v4.1.
>>>
>>> $ $MODULES_CMD python autoinit
>>> import os
>>> import re, subprocess
>>> def module(*arguments):
>>>    ns = {}
>>>    exec(subprocess.Popen(['/usr/bin/tclsh',
>>> '/usr/share/Modules/libexec/modulecmd.tcl', 'python'] + list(arguments),
>>> stdout=subprocess.PIPE).communicate()[0], ns)
>>>    if '_mlstatus' in ns:
>>>       _mlstatus = ns['_mlstatus']
>>>    else:
>>>       _mlstatus = True
>>>    return _mlstatus
>>>
>>> With the code example you gave, the module function used is the one
>>> defined in a sh shell session (due to the shell=True parameter of
>>> subprocess.run). Usually the module shell function returns the output on
>>> stdout (stderr messages are redirected to stdout after the evaluation of
>>> the shell code generated on stdout).
>>>
>>> Regards,
>>> Xavier
>>>
>>> Le mer. 23 mars 2022 à 14:26, Jonathan Buck <jon...@gm...> a
>>> écrit :
>>>
>>>> Hi Paul,
>>>>
>>>> Thanks for the response!  I took a look at this and spoke with a
>>>> colleague, but exit status is hard to grab from the module() method within
>>>> python.  The only way we were able to pull this off was to ignore the
>>>> suggested python exec() statement found in the documentation and rather
>>>> create our own popen() statement.  This allowed us to then check the status
>>>> of that process.  What I don't like about this solution is that we're
>>>> depending on the originating shell to have properly sourced the module
>>>> environment, so that we can execute these 'os' commands.  Here's a snippet
>>>> of the code my colleague came up with:
>>>>
>>>> for mod in modules_to_test:    proc = subprocess.run(f"module use {mod_path};module load {mod}",            stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True,            universal_newlines=True)
>>>>     print("", "########", f"Test: {mod}", "########", sep="\n")    # Just FYI to see the attributes of the object    # print(proc)    print(f"stdout: {proc.stdout}")    print(f"stderr: {proc.stderr}")    print(f"return code: {proc.returncode}")
>>>>
>>>> We Found that all output goes to proc.stdout rather than any errors
>>>> going to proc.stderr, regardless of a return code being 1 or 0.  The return
>>>> code is the winner here. So, this works but I'm calling it a dirty solution
>>>> because I think that once I automate this and try to execute remotely on
>>>> our clusters, we'll lose the properly sourced environment.
>>>>
>>>> On Tue, Mar 22, 2022 at 4:19 PM Paul Markfort <pau...@gm...>
>>>> wrote:
>>>>
>>>>> If you are just trying to, determine if there was an error thrown by
>>>>> the module command,
>>>>> Just test the exit status:
>>>>>
>>>>> EXIT STATUS
>>>>>         The module command exits with 0 if its execution succeed.
>>>>> Otherwise  1
>>>>>         is returned.
>>>>>
>>>>>
>>>>> Keep in mind that every call to the module command redirects
>>>>> sys.stdout so that the output is evaluated by python (so you probably can't
>>>>> capture that).
>>>>> And sys.stderr redirection by a module command might depend on what
>>>>> stderr is connected to.
>>>>>
>>>>> Last - stderr and stdout are buffered IO, so you may also have to do a
>>>>> flush before reading them.
>>>>>
>>>>>
>>>>> I should preface the below with, "I haven't coded in python":
>>>>>
>>>>> Maybe you need to test your code with no modules involved first.
>>>>> You might want to read this (seems like stringIO() may have some
>>>>> limitations).
>>>>>
>>>>> https://stackoverflow.com/questions/1218933/can-i-redirect-the-stdout-into-some-sort-of-string-buffer
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 2022-03-22 09:18 AM, Jonathan Buck wrote:
>>>>> > Hi Jacques,
>>>>> >
>>>>> > Thanks for the quick reply!  I have successfully use the module()
>>>>> command
>>>>> > to setup environments without issue.  What I'm hoping to do is
>>>>> capture any
>>>>> > stderr messages to a variable.  I have several use cases for this,
>>>>> most
>>>>> > importantly I'm attempting to automate checks against our hundreds of
>>>>> > modules to determine if any of them are broken when loaded.  I'm
>>>>> unable to
>>>>> > redirect stdout/stderr in a manner that I've used before which I
>>>>> believe is
>>>>> > due to the command being a subprocess.  I'm not a python guru, but
>>>>> the
>>>>> > popen statement declares " stdout=subprocess.PIPE" which I thought
>>>>> was the
>>>>> > right way to get the stdout returned to the originating process.
>>>>> >
>>>>> > All that being said, here's my test script:
>>>>> >
>>>>> > import os
>>>>> > import sys
>>>>> > from io import StringIO
>>>>> >
>>>>> > # Setup variables to hold stdout/stderr
>>>>> > result_stdout = StringIO()
>>>>> > result_stderr = StringIO()
>>>>> > sys.stdout = result_stdout
>>>>> > sys.stderr = result_stderr
>>>>> >
>>>>> > # setup environment module
>>>>> > exec(open('/p/app/Modules/4.7.1/init/python.py').read())
>>>>> >
>>>>> > # run
>>>>> > print("----- TRY RUNNING MODULE COMMAND -----")
>>>>> > module('--version')
>>>>> > print("----- DONE RUNNING MODULE COMMAND -----")
>>>>> > module_stdout = result_stdout.getvalue()
>>>>> > module_stderr = result_stderr.getvalue()
>>>>> >
>>>>> > # reset stdout/stderr to defaults
>>>>> > sys.stdout = sys.__stdout__
>>>>> > sys.stderr = sys.__stderr__
>>>>> >
>>>>> > # print out what was captured to the variables
>>>>> > print("===== TRY PRINTING STDOUT =====")
>>>>> > sys.stdout.write(module_stdout)
>>>>> > print("===== DONE PRINTING STDOUT =====")
>>>>> >
>>>>> > print("***** TRY PRINTING STDERR *****")
>>>>> > sys.stderr.write(module_stderr)
>>>>> > print("***** DONE PRINTING STDERR *****")
>>>>> >
>>>>> >
>>>>> > As you can see, it's simple.  I make sure to reset stdout/stderr to
>>>>> > defaults, however these are my results:
>>>>> >
>>>>> > ~> python3 module_test.py
>>>>> > Modules Release 4.7.1 (2021-04-06)
>>>>> > ===== TRY PRINTING JSON STDOUT =====
>>>>> > ----- TRY RUNNING MODULE COMMAND -----
>>>>> > ----- DONE RUNNING MODULE COMMAND -----
>>>>> > ===== DONE PRINTING JSON STDOUT =====
>>>>> > ***** TRY PRINTING JSON STDERR *****
>>>>> > ***** DONE PRINTING JSON STDERR *****
>>>>> > ~>
>>>>> >
>>>>> > This shows that my stdout redirect is capturing the 'print'
>>>>> statements
>>>>> > for running the module command.  However the output for the module
>>>>> > command prints at the very beginning and doesn't show up.  Any
>>>>> > thoughts?  I found a fancy way to do the same thing in bash, and I
>>>>> > proved that in bash the same output is going to stdout.
>>>>> >
>>>>> >
>>>>> > Very Respectfully,
>>>>> >
>>>>> > Jonathan
>>>>> >
>>>>> >
>>>>> > On Tue, Mar 22, 2022 at 8:06 AM Contact <dv...@cr...> wrote:
>>>>> >
>>>>> >> Hello,
>>>>> >>
>>>>> >> In my case I used modules from python with the following logic:
>>>>> >>
>>>>> >> import os
>>>>> >> exec(open(/home/jrp/modules/init/python.py')).read())
>>>>> >> module('avail')
>>>>> >> module('load', 'gcc/7')
>>>>> >> module('list')
>>>>> >>
>>>>> >>  From the above example,
>>>>> >>
>>>>> >>     - exec call will allow me to use modules from python itself and
>>>>> define
>>>>> >>     module function
>>>>> >>     - Each module function invocation will be redirected to module
>>>>> to
>>>>> >>     perform required action
>>>>> >>
>>>>> >> As reminder, the content of stdout is used by modules to setup all
>>>>> >> required environment functions, functions/alias (if supported by
>>>>> current
>>>>> >> shell), ...
>>>>> >>
>>>>> >> I hope it will help you.
>>>>> >>
>>>>> >> Jacques
>>>>> >>
>>>>> >>
>>>>> >> Le 21/03/2022 à 20:09, Jonathan Buck a écrit :
>>>>> >>
>>>>> >> Hi,
>>>>> >>
>>>>> >> I'm attempting to capture and act on the stdout and stderr of the
>>>>> module
>>>>> >> command.  The python implementation appears to be a subprocess for
>>>>> which I
>>>>> >> wasn't able to redirect with io.StringIO() as I have with other
>>>>> output
>>>>> >> capture.
>>>>> >>
>>>>> >> I attempted to look through the code, which I admit I don't fully
>>>>> >> understand and appears to have a one-liner style of executing the
>>>>> >> subprocess.  I'm writing code that integrates with a production
>>>>> system, so
>>>>> >> I'm not able to modify the python.py code.
>>>>> >>
>>>>> >> Does anyone know how else this might be accomplished?  Or should I
>>>>> make
>>>>> >> this a feature request?
>>>>> >>
>>>>> >> Thanks for your help,
>>>>> >> Jonathan
>>>>>
>>>> _______________________________________________
> Modules-interest mailing list
> Mod...@li...
> https://lists.sourceforge.net/lists/listinfo/modules-interest
>

Re: [Modules] Capture python stdout and stderr

[Xavier D. <xav...@gm...>]

Le mer. 23 mars 2022 à 16:31, Jonathan Buck <jon...@gm...> a
écrit :

>   Hi Xavier,
>
> This is great!  Thanks for showing me how to expose the underlying
> function!  One thing to note is that everything returned comes through
> stderr.  So, as you were saying, I wasn't able to trap anything because
> everything was coming through stderr.
>
> In my testing, _mlstatus is returned by stdout.  Are there other values
> that might be returned by stdout other than "_mlstatus = False" ?
>
> Am I right in assuming '_mlstatus' is the return code of the tcl execution?
>

_mlstatus is the variable that hold the return code of the evaluation. On
stdout also goes all the environment changes implied by modulefile
evaluation.


> The only problem I still have is that I can't seem to capture that stderr
> output to a variable.  Currently, it goes to the console, no matter what I
> do.  I'm going to play around with it and see if this is because of the
> exec() or the popen() statements.  Ideally, I would like to capture the
> output when the _mlstatus is False.  For my use case, This feature would
> allow me to compile a list of my 'bad' modules and their associated output.
>
> I've been playing with your initial code this morning, teasing it apart
> and designing a return object containing the output and the return code.
> This might not be the preferred methodology, but either way I'll report
> back with what I find.
>

With the code I proposed this morning, you should be able to get the
content from sys.stderr. On my side, I have run this proposed code in the
python script you provided and I correctly got that stderr output between
the TRY/DONE PRINTING STDERR lines.

Regards,
Xavier



> Thanks again and with all respect,
> Jonathan
>
>
> On Wed, Mar 23, 2022 at 11:14 AM Xavier Delaruelle <
> xav...@gm...> wrote:
>
>> Hi Jonathan,
>>
>> The exit status should be easy to grab from the module() python function.
>> This function basically returns this exit status as a boolean. I paste
>> below the current python code of the module() definition in python. It has
>> not changed a lot since Modules v4.1.
>>
>> $ $MODULES_CMD python autoinit
>> import os
>> import re, subprocess
>> def module(*arguments):
>>    ns = {}
>>    exec(subprocess.Popen(['/usr/bin/tclsh',
>> '/usr/share/Modules/libexec/modulecmd.tcl', 'python'] + list(arguments),
>> stdout=subprocess.PIPE).communicate()[0], ns)
>>    if '_mlstatus' in ns:
>>       _mlstatus = ns['_mlstatus']
>>    else:
>>       _mlstatus = True
>>    return _mlstatus
>>
>> With the code example you gave, the module function used is the one
>> defined in a sh shell session (due to the shell=True parameter of
>> subprocess.run). Usually the module shell function returns the output on
>> stdout (stderr messages are redirected to stdout after the evaluation of
>> the shell code generated on stdout).
>>
>> Regards,
>> Xavier
>>
>> Le mer. 23 mars 2022 à 14:26, Jonathan Buck <jon...@gm...> a
>> écrit :
>>
>>> Hi Paul,
>>>
>>> Thanks for the response!  I took a look at this and spoke with a
>>> colleague, but exit status is hard to grab from the module() method within
>>> python.  The only way we were able to pull this off was to ignore the
>>> suggested python exec() statement found in the documentation and rather
>>> create our own popen() statement.  This allowed us to then check the status
>>> of that process.  What I don't like about this solution is that we're
>>> depending on the originating shell to have properly sourced the module
>>> environment, so that we can execute these 'os' commands.  Here's a snippet
>>> of the code my colleague came up with:
>>>
>>> for mod in modules_to_test:    proc = subprocess.run(f"module use {mod_path};module load {mod}",            stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True,            universal_newlines=True)
>>>     print("", "########", f"Test: {mod}", "########", sep="\n")    # Just FYI to see the attributes of the object    # print(proc)    print(f"stdout: {proc.stdout}")    print(f"stderr: {proc.stderr}")    print(f"return code: {proc.returncode}")
>>>
>>> We Found that all output goes to proc.stdout rather than any errors
>>> going to proc.stderr, regardless of a return code being 1 or 0.  The return
>>> code is the winner here. So, this works but I'm calling it a dirty solution
>>> because I think that once I automate this and try to execute remotely on
>>> our clusters, we'll lose the properly sourced environment.
>>>
>>> On Tue, Mar 22, 2022 at 4:19 PM Paul Markfort <pau...@gm...>
>>> wrote:
>>>
>>>> If you are just trying to, determine if there was an error thrown by
>>>> the module command,
>>>> Just test the exit status:
>>>>
>>>> EXIT STATUS
>>>>         The module command exits with 0 if its execution succeed.
>>>> Otherwise  1
>>>>         is returned.
>>>>
>>>>
>>>> Keep in mind that every call to the module command redirects sys.stdout
>>>> so that the output is evaluated by python (so you probably can't capture
>>>> that).
>>>> And sys.stderr redirection by a module command might depend on what
>>>> stderr is connected to.
>>>>
>>>> Last - stderr and stdout are buffered IO, so you may also have to do a
>>>> flush before reading them.
>>>>
>>>>
>>>> I should preface the below with, "I haven't coded in python":
>>>>
>>>> Maybe you need to test your code with no modules involved first.
>>>> You might want to read this (seems like stringIO() may have some
>>>> limitations).
>>>>
>>>> https://stackoverflow.com/questions/1218933/can-i-redirect-the-stdout-into-some-sort-of-string-buffer
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On 2022-03-22 09:18 AM, Jonathan Buck wrote:
>>>> > Hi Jacques,
>>>> >
>>>> > Thanks for the quick reply!  I have successfully use the module()
>>>> command
>>>> > to setup environments without issue.  What I'm hoping to do is
>>>> capture any
>>>> > stderr messages to a variable.  I have several use cases for this,
>>>> most
>>>> > importantly I'm attempting to automate checks against our hundreds of
>>>> > modules to determine if any of them are broken when loaded.  I'm
>>>> unable to
>>>> > redirect stdout/stderr in a manner that I've used before which I
>>>> believe is
>>>> > due to the command being a subprocess.  I'm not a python guru, but the
>>>> > popen statement declares " stdout=subprocess.PIPE" which I thought
>>>> was the
>>>> > right way to get the stdout returned to the originating process.
>>>> >
>>>> > All that being said, here's my test script:
>>>> >
>>>> > import os
>>>> > import sys
>>>> > from io import StringIO
>>>> >
>>>> > # Setup variables to hold stdout/stderr
>>>> > result_stdout = StringIO()
>>>> > result_stderr = StringIO()
>>>> > sys.stdout = result_stdout
>>>> > sys.stderr = result_stderr
>>>> >
>>>> > # setup environment module
>>>> > exec(open('/p/app/Modules/4.7.1/init/python.py').read())
>>>> >
>>>> > # run
>>>> > print("----- TRY RUNNING MODULE COMMAND -----")
>>>> > module('--version')
>>>> > print("----- DONE RUNNING MODULE COMMAND -----")
>>>> > module_stdout = result_stdout.getvalue()
>>>> > module_stderr = result_stderr.getvalue()
>>>> >
>>>> > # reset stdout/stderr to defaults
>>>> > sys.stdout = sys.__stdout__
>>>> > sys.stderr = sys.__stderr__
>>>> >
>>>> > # print out what was captured to the variables
>>>> > print("===== TRY PRINTING STDOUT =====")
>>>> > sys.stdout.write(module_stdout)
>>>> > print("===== DONE PRINTING STDOUT =====")
>>>> >
>>>> > print("***** TRY PRINTING STDERR *****")
>>>> > sys.stderr.write(module_stderr)
>>>> > print("***** DONE PRINTING STDERR *****")
>>>> >
>>>> >
>>>> > As you can see, it's simple.  I make sure to reset stdout/stderr to
>>>> > defaults, however these are my results:
>>>> >
>>>> > ~> python3 module_test.py
>>>> > Modules Release 4.7.1 (2021-04-06)
>>>> > ===== TRY PRINTING JSON STDOUT =====
>>>> > ----- TRY RUNNING MODULE COMMAND -----
>>>> > ----- DONE RUNNING MODULE COMMAND -----
>>>> > ===== DONE PRINTING JSON STDOUT =====
>>>> > ***** TRY PRINTING JSON STDERR *****
>>>> > ***** DONE PRINTING JSON STDERR *****
>>>> > ~>
>>>> >
>>>> > This shows that my stdout redirect is capturing the 'print' statements
>>>> > for running the module command.  However the output for the module
>>>> > command prints at the very beginning and doesn't show up.  Any
>>>> > thoughts?  I found a fancy way to do the same thing in bash, and I
>>>> > proved that in bash the same output is going to stdout.
>>>> >
>>>> >
>>>> > Very Respectfully,
>>>> >
>>>> > Jonathan
>>>> >
>>>> >
>>>> > On Tue, Mar 22, 2022 at 8:06 AM Contact <dv...@cr...> wrote:
>>>> >
>>>> >> Hello,
>>>> >>
>>>> >> In my case I used modules from python with the following logic:
>>>> >>
>>>> >> import os
>>>> >> exec(open(/home/jrp/modules/init/python.py')).read())
>>>> >> module('avail')
>>>> >> module('load', 'gcc/7')
>>>> >> module('list')
>>>> >>
>>>> >>  From the above example,
>>>> >>
>>>> >>     - exec call will allow me to use modules from python itself and
>>>> define
>>>> >>     module function
>>>> >>     - Each module function invocation will be redirected to module to
>>>> >>     perform required action
>>>> >>
>>>> >> As reminder, the content of stdout is used by modules to setup all
>>>> >> required environment functions, functions/alias (if supported by
>>>> current
>>>> >> shell), ...
>>>> >>
>>>> >> I hope it will help you.
>>>> >>
>>>> >> Jacques
>>>> >>
>>>> >>
>>>> >> Le 21/03/2022 à 20:09, Jonathan Buck a écrit :
>>>> >>
>>>> >> Hi,
>>>> >>
>>>> >> I'm attempting to capture and act on the stdout and stderr of the
>>>> module
>>>> >> command.  The python implementation appears to be a subprocess for
>>>> which I
>>>> >> wasn't able to redirect with io.StringIO() as I have with other
>>>> output
>>>> >> capture.
>>>> >>
>>>> >> I attempted to look through the code, which I admit I don't fully
>>>> >> understand and appears to have a one-liner style of executing the
>>>> >> subprocess.  I'm writing code that integrates with a production
>>>> system, so
>>>> >> I'm not able to modify the python.py code.
>>>> >>
>>>> >> Does anyone know how else this might be accomplished?  Or should I
>>>> make
>>>> >> this a feature request?
>>>> >>
>>>> >> Thanks for your help,
>>>> >> Jonathan
>>>>
>>>

Re: [Modules] Capture python stdout and stderr

[Jonathan B. <jon...@gm...>]

  Hi Xavier,

This is great!  Thanks for showing me how to expose the underlying
function!  One thing to note is that everything returned comes through
stderr.  So, as you were saying, I wasn't able to trap anything because
everything was coming through stderr.

In my testing, _mlstatus is returned by stdout.  Are there other values
that might be returned by stdout other than "_mlstatus = False" ?

Am I right in assuming '_mlstatus' is the return code of the tcl execution?

The only problem I still have is that I can't seem to capture that stderr
output to a variable.  Currently, it goes to the console, no matter what I
do.  I'm going to play around with it and see if this is because of the
exec() or the popen() statements.  Ideally, I would like to capture the
output when the _mlstatus is False.  For my use case, This feature would
allow me to compile a list of my 'bad' modules and their associated output.

I've been playing with your initial code this morning, teasing it apart and
designing a return object containing the output and the return code.  This
might not be the preferred methodology, but either way I'll report back
with what I find.

Thanks again and with all respect,
Jonathan


On Wed, Mar 23, 2022 at 11:14 AM Xavier Delaruelle <
xav...@gm...> wrote:

> Hi Jonathan,
>
> The exit status should be easy to grab from the module() python function.
> This function basically returns this exit status as a boolean. I paste
> below the current python code of the module() definition in python. It has
> not changed a lot since Modules v4.1.
>
> $ $MODULES_CMD python autoinit
> import os
> import re, subprocess
> def module(*arguments):
>    ns = {}
>    exec(subprocess.Popen(['/usr/bin/tclsh',
> '/usr/share/Modules/libexec/modulecmd.tcl', 'python'] + list(arguments),
> stdout=subprocess.PIPE).communicate()[0], ns)
>    if '_mlstatus' in ns:
>       _mlstatus = ns['_mlstatus']
>    else:
>       _mlstatus = True
>    return _mlstatus
>
> With the code example you gave, the module function used is the one
> defined in a sh shell session (due to the shell=True parameter of
> subprocess.run). Usually the module shell function returns the output on
> stdout (stderr messages are redirected to stdout after the evaluation of
> the shell code generated on stdout).
>
> Regards,
> Xavier
>
> Le mer. 23 mars 2022 à 14:26, Jonathan Buck <jon...@gm...> a
> écrit :
>
>> Hi Paul,
>>
>> Thanks for the response!  I took a look at this and spoke with a
>> colleague, but exit status is hard to grab from the module() method within
>> python.  The only way we were able to pull this off was to ignore the
>> suggested python exec() statement found in the documentation and rather
>> create our own popen() statement.  This allowed us to then check the status
>> of that process.  What I don't like about this solution is that we're
>> depending on the originating shell to have properly sourced the module
>> environment, so that we can execute these 'os' commands.  Here's a snippet
>> of the code my colleague came up with:
>>
>> for mod in modules_to_test:    proc = subprocess.run(f"module use {mod_path};module load {mod}",            stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True,            universal_newlines=True)
>>     print("", "########", f"Test: {mod}", "########", sep="\n")    # Just FYI to see the attributes of the object    # print(proc)    print(f"stdout: {proc.stdout}")    print(f"stderr: {proc.stderr}")    print(f"return code: {proc.returncode}")
>>
>> We Found that all output goes to proc.stdout rather than any errors going
>> to proc.stderr, regardless of a return code being 1 or 0.  The return code
>> is the winner here. So, this works but I'm calling it a dirty solution
>> because I think that once I automate this and try to execute remotely on
>> our clusters, we'll lose the properly sourced environment.
>>
>> On Tue, Mar 22, 2022 at 4:19 PM Paul Markfort <pau...@gm...>
>> wrote:
>>
>>> If you are just trying to, determine if there was an error thrown by the
>>> module command,
>>> Just test the exit status:
>>>
>>> EXIT STATUS
>>>         The module command exits with 0 if its execution succeed.
>>> Otherwise  1
>>>         is returned.
>>>
>>>
>>> Keep in mind that every call to the module command redirects sys.stdout
>>> so that the output is evaluated by python (so you probably can't capture
>>> that).
>>> And sys.stderr redirection by a module command might depend on what
>>> stderr is connected to.
>>>
>>> Last - stderr and stdout are buffered IO, so you may also have to do a
>>> flush before reading them.
>>>
>>>
>>> I should preface the below with, "I haven't coded in python":
>>>
>>> Maybe you need to test your code with no modules involved first.
>>> You might want to read this (seems like stringIO() may have some
>>> limitations).
>>>
>>> https://stackoverflow.com/questions/1218933/can-i-redirect-the-stdout-into-some-sort-of-string-buffer
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On 2022-03-22 09:18 AM, Jonathan Buck wrote:
>>> > Hi Jacques,
>>> >
>>> > Thanks for the quick reply!  I have successfully use the module()
>>> command
>>> > to setup environments without issue.  What I'm hoping to do is capture
>>> any
>>> > stderr messages to a variable.  I have several use cases for this, most
>>> > importantly I'm attempting to automate checks against our hundreds of
>>> > modules to determine if any of them are broken when loaded.  I'm
>>> unable to
>>> > redirect stdout/stderr in a manner that I've used before which I
>>> believe is
>>> > due to the command being a subprocess.  I'm not a python guru, but the
>>> > popen statement declares " stdout=subprocess.PIPE" which I thought was
>>> the
>>> > right way to get the stdout returned to the originating process.
>>> >
>>> > All that being said, here's my test script:
>>> >
>>> > import os
>>> > import sys
>>> > from io import StringIO
>>> >
>>> > # Setup variables to hold stdout/stderr
>>> > result_stdout = StringIO()
>>> > result_stderr = StringIO()
>>> > sys.stdout = result_stdout
>>> > sys.stderr = result_stderr
>>> >
>>> > # setup environment module
>>> > exec(open('/p/app/Modules/4.7.1/init/python.py').read())
>>> >
>>> > # run
>>> > print("----- TRY RUNNING MODULE COMMAND -----")
>>> > module('--version')
>>> > print("----- DONE RUNNING MODULE COMMAND -----")
>>> > module_stdout = result_stdout.getvalue()
>>> > module_stderr = result_stderr.getvalue()
>>> >
>>> > # reset stdout/stderr to defaults
>>> > sys.stdout = sys.__stdout__
>>> > sys.stderr = sys.__stderr__
>>> >
>>> > # print out what was captured to the variables
>>> > print("===== TRY PRINTING STDOUT =====")
>>> > sys.stdout.write(module_stdout)
>>> > print("===== DONE PRINTING STDOUT =====")
>>> >
>>> > print("***** TRY PRINTING STDERR *****")
>>> > sys.stderr.write(module_stderr)
>>> > print("***** DONE PRINTING STDERR *****")
>>> >
>>> >
>>> > As you can see, it's simple.  I make sure to reset stdout/stderr to
>>> > defaults, however these are my results:
>>> >
>>> > ~> python3 module_test.py
>>> > Modules Release 4.7.1 (2021-04-06)
>>> > ===== TRY PRINTING JSON STDOUT =====
>>> > ----- TRY RUNNING MODULE COMMAND -----
>>> > ----- DONE RUNNING MODULE COMMAND -----
>>> > ===== DONE PRINTING JSON STDOUT =====
>>> > ***** TRY PRINTING JSON STDERR *****
>>> > ***** DONE PRINTING JSON STDERR *****
>>> > ~>
>>> >
>>> > This shows that my stdout redirect is capturing the 'print' statements
>>> > for running the module command.  However the output for the module
>>> > command prints at the very beginning and doesn't show up.  Any
>>> > thoughts?  I found a fancy way to do the same thing in bash, and I
>>> > proved that in bash the same output is going to stdout.
>>> >
>>> >
>>> > Very Respectfully,
>>> >
>>> > Jonathan
>>> >
>>> >
>>> > On Tue, Mar 22, 2022 at 8:06 AM Contact <dv...@cr...> wrote:
>>> >
>>> >> Hello,
>>> >>
>>> >> In my case I used modules from python with the following logic:
>>> >>
>>> >> import os
>>> >> exec(open(/home/jrp/modules/init/python.py')).read())
>>> >> module('avail')
>>> >> module('load', 'gcc/7')
>>> >> module('list')
>>> >>
>>> >>  From the above example,
>>> >>
>>> >>     - exec call will allow me to use modules from python itself and
>>> define
>>> >>     module function
>>> >>     - Each module function invocation will be redirected to module to
>>> >>     perform required action
>>> >>
>>> >> As reminder, the content of stdout is used by modules to setup all
>>> >> required environment functions, functions/alias (if supported by
>>> current
>>> >> shell), ...
>>> >>
>>> >> I hope it will help you.
>>> >>
>>> >> Jacques
>>> >>
>>> >>
>>> >> Le 21/03/2022 à 20:09, Jonathan Buck a écrit :
>>> >>
>>> >> Hi,
>>> >>
>>> >> I'm attempting to capture and act on the stdout and stderr of the
>>> module
>>> >> command.  The python implementation appears to be a subprocess for
>>> which I
>>> >> wasn't able to redirect with io.StringIO() as I have with other output
>>> >> capture.
>>> >>
>>> >> I attempted to look through the code, which I admit I don't fully
>>> >> understand and appears to have a one-liner style of executing the
>>> >> subprocess.  I'm writing code that integrates with a production
>>> system, so
>>> >> I'm not able to modify the python.py code.
>>> >>
>>> >> Does anyone know how else this might be accomplished?  Or should I
>>> make
>>> >> this a feature request?
>>> >>
>>> >> Thanks for your help,
>>> >> Jonathan
>>> >>
>>> >>
>>> >> _______________________________________________
>>> >> Modules-interest mailing listModules-interest
>>> @lists.sourceforge.nethttps://
>>> lists.sourceforge.net/lists/listinfo/modules-interest
>>> >>
>>> >>
>>> >
>>> >
>>> >
>>> > _______________________________________________
>>> > Modules-interest mailing list
>>> > Mod...@li...
>>> > https://lists.sourceforge.net/lists/listinfo/modules-interest
>>> >
>>>
>>> --
>>> --------------------------------------------------------
>>> The views and opinions expressed above are strictly
>>> those of the author(s).  The content of this message has
>>> not been reviewed nor approved by any entity whatsoever.
>>> --------------------------------------------------------
>>> Paul FM         Info: http://paulfm.com/~paulfm/
>>> --------------------------------------------------------
>>>
>>>
>>> _______________________________________________
>>> Modules-interest mailing list
>>> Mod...@li...
>>> https://lists.sourceforge.net/lists/listinfo/modules-interest
>>>
>> _______________________________________________
>> Modules-interest mailing list
>> Mod...@li...
>> https://lists.sourceforge.net/lists/listinfo/modules-interest
>>
> _______________________________________________
> Modules-interest mailing list
> Mod...@li...
> https://lists.sourceforge.net/lists/listinfo/modules-interest
>

Re: [Modules] Capture python stdout and stderr

[Xavier D. <xav...@gm...>]

Hi Jonathan,

The exit status should be easy to grab from the module() python function.
This function basically returns this exit status as a boolean. I paste
below the current python code of the module() definition in python. It has
not changed a lot since Modules v4.1.

$ $MODULES_CMD python autoinit
import os
import re, subprocess
def module(*arguments):
   ns = {}
   exec(subprocess.Popen(['/usr/bin/tclsh',
'/usr/share/Modules/libexec/modulecmd.tcl', 'python'] + list(arguments),
stdout=subprocess.PIPE).communicate()[0], ns)
   if '_mlstatus' in ns:
      _mlstatus = ns['_mlstatus']
   else:
      _mlstatus = True
   return _mlstatus

With the code example you gave, the module function used is the one defined
in a sh shell session (due to the shell=True parameter of subprocess.run).
Usually the module shell function returns the output on stdout (stderr
messages are redirected to stdout after the evaluation of the shell code
generated on stdout).

Regards,
Xavier

Le mer. 23 mars 2022 à 14:26, Jonathan Buck <jon...@gm...> a
écrit :

> Hi Paul,
>
> Thanks for the response!  I took a look at this and spoke with a
> colleague, but exit status is hard to grab from the module() method within
> python.  The only way we were able to pull this off was to ignore the
> suggested python exec() statement found in the documentation and rather
> create our own popen() statement.  This allowed us to then check the status
> of that process.  What I don't like about this solution is that we're
> depending on the originating shell to have properly sourced the module
> environment, so that we can execute these 'os' commands.  Here's a snippet
> of the code my colleague came up with:
>
> for mod in modules_to_test:    proc = subprocess.run(f"module use {mod_path};module load {mod}",            stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True,            universal_newlines=True)
>     print("", "########", f"Test: {mod}", "########", sep="\n")    # Just FYI to see the attributes of the object    # print(proc)    print(f"stdout: {proc.stdout}")    print(f"stderr: {proc.stderr}")    print(f"return code: {proc.returncode}")
>
> We Found that all output goes to proc.stdout rather than any errors going
> to proc.stderr, regardless of a return code being 1 or 0.  The return code
> is the winner here. So, this works but I'm calling it a dirty solution
> because I think that once I automate this and try to execute remotely on
> our clusters, we'll lose the properly sourced environment.
>
> On Tue, Mar 22, 2022 at 4:19 PM Paul Markfort <pau...@gm...> wrote:
>
>> If you are just trying to, determine if there was an error thrown by the
>> module command,
>> Just test the exit status:
>>
>> EXIT STATUS
>>         The module command exits with 0 if its execution succeed.
>> Otherwise  1
>>         is returned.
>>
>>
>> Keep in mind that every call to the module command redirects sys.stdout
>> so that the output is evaluated by python (so you probably can't capture
>> that).
>> And sys.stderr redirection by a module command might depend on what
>> stderr is connected to.
>>
>> Last - stderr and stdout are buffered IO, so you may also have to do a
>> flush before reading them.
>>
>>
>> I should preface the below with, "I haven't coded in python":
>>
>> Maybe you need to test your code with no modules involved first.
>> You might want to read this (seems like stringIO() may have some
>> limitations).
>>
>> https://stackoverflow.com/questions/1218933/can-i-redirect-the-stdout-into-some-sort-of-string-buffer
>>
>>
>>
>>
>>
>>
>>
>> On 2022-03-22 09:18 AM, Jonathan Buck wrote:
>> > Hi Jacques,
>> >
>> > Thanks for the quick reply!  I have successfully use the module()
>> command
>> > to setup environments without issue.  What I'm hoping to do is capture
>> any
>> > stderr messages to a variable.  I have several use cases for this, most
>> > importantly I'm attempting to automate checks against our hundreds of
>> > modules to determine if any of them are broken when loaded.  I'm unable
>> to
>> > redirect stdout/stderr in a manner that I've used before which I
>> believe is
>> > due to the command being a subprocess.  I'm not a python guru, but the
>> > popen statement declares " stdout=subprocess.PIPE" which I thought was
>> the
>> > right way to get the stdout returned to the originating process.
>> >
>> > All that being said, here's my test script:
>> >
>> > import os
>> > import sys
>> > from io import StringIO
>> >
>> > # Setup variables to hold stdout/stderr
>> > result_stdout = StringIO()
>> > result_stderr = StringIO()
>> > sys.stdout = result_stdout
>> > sys.stderr = result_stderr
>> >
>> > # setup environment module
>> > exec(open('/p/app/Modules/4.7.1/init/python.py').read())
>> >
>> > # run
>> > print("----- TRY RUNNING MODULE COMMAND -----")
>> > module('--version')
>> > print("----- DONE RUNNING MODULE COMMAND -----")
>> > module_stdout = result_stdout.getvalue()
>> > module_stderr = result_stderr.getvalue()
>> >
>> > # reset stdout/stderr to defaults
>> > sys.stdout = sys.__stdout__
>> > sys.stderr = sys.__stderr__
>> >
>> > # print out what was captured to the variables
>> > print("===== TRY PRINTING STDOUT =====")
>> > sys.stdout.write(module_stdout)
>> > print("===== DONE PRINTING STDOUT =====")
>> >
>> > print("***** TRY PRINTING STDERR *****")
>> > sys.stderr.write(module_stderr)
>> > print("***** DONE PRINTING STDERR *****")
>> >
>> >
>> > As you can see, it's simple.  I make sure to reset stdout/stderr to
>> > defaults, however these are my results:
>> >
>> > ~> python3 module_test.py
>> > Modules Release 4.7.1 (2021-04-06)
>> > ===== TRY PRINTING JSON STDOUT =====
>> > ----- TRY RUNNING MODULE COMMAND -----
>> > ----- DONE RUNNING MODULE COMMAND -----
>> > ===== DONE PRINTING JSON STDOUT =====
>> > ***** TRY PRINTING JSON STDERR *****
>> > ***** DONE PRINTING JSON STDERR *****
>> > ~>
>> >
>> > This shows that my stdout redirect is capturing the 'print' statements
>> > for running the module command.  However the output for the module
>> > command prints at the very beginning and doesn't show up.  Any
>> > thoughts?  I found a fancy way to do the same thing in bash, and I
>> > proved that in bash the same output is going to stdout.
>> >
>> >
>> > Very Respectfully,
>> >
>> > Jonathan
>> >
>> >
>> > On Tue, Mar 22, 2022 at 8:06 AM Contact <dv...@cr...> wrote:
>> >
>> >> Hello,
>> >>
>> >> In my case I used modules from python with the following logic:
>> >>
>> >> import os
>> >> exec(open(/home/jrp/modules/init/python.py')).read())
>> >> module('avail')
>> >> module('load', 'gcc/7')
>> >> module('list')
>> >>
>> >>  From the above example,
>> >>
>> >>     - exec call will allow me to use modules from python itself and
>> define
>> >>     module function
>> >>     - Each module function invocation will be redirected to module to
>> >>     perform required action
>> >>
>> >> As reminder, the content of stdout is used by modules to setup all
>> >> required environment functions, functions/alias (if supported by
>> current
>> >> shell), ...
>> >>
>> >> I hope it will help you.
>> >>
>> >> Jacques
>> >>
>> >>
>> >> Le 21/03/2022 à 20:09, Jonathan Buck a écrit :
>> >>
>> >> Hi,
>> >>
>> >> I'm attempting to capture and act on the stdout and stderr of the
>> module
>> >> command.  The python implementation appears to be a subprocess for
>> which I
>> >> wasn't able to redirect with io.StringIO() as I have with other output
>> >> capture.
>> >>
>> >> I attempted to look through the code, which I admit I don't fully
>> >> understand and appears to have a one-liner style of executing the
>> >> subprocess.  I'm writing code that integrates with a production
>> system, so
>> >> I'm not able to modify the python.py code.
>> >>
>> >> Does anyone know how else this might be accomplished?  Or should I make
>> >> this a feature request?
>> >>
>> >> Thanks for your help,
>> >> Jonathan
>> >>
>> >>
>> >> _______________________________________________
>> >> Modules-interest mailing listModules-interest
>> @lists.sourceforge.nethttps://
>> lists.sourceforge.net/lists/listinfo/modules-interest
>> >>
>> >>
>> >
>> >
>> >
>> > _______________________________________________
>> > Modules-interest mailing list
>> > Mod...@li...
>> > https://lists.sourceforge.net/lists/listinfo/modules-interest
>> >
>>
>> --
>> --------------------------------------------------------
>> The views and opinions expressed above are strictly
>> those of the author(s).  The content of this message has
>> not been reviewed nor approved by any entity whatsoever.
>> --------------------------------------------------------
>> Paul FM         Info: http://paulfm.com/~paulfm/
>> --------------------------------------------------------
>>
>>
>> _______________________________________________
>> Modules-interest mailing list
>> Mod...@li...
>> https://lists.sourceforge.net/lists/listinfo/modules-interest
>>
> _______________________________________________
> Modules-interest mailing list
> Mod...@li...
> https://lists.sourceforge.net/lists/listinfo/modules-interest
>

Re: [Modules] Capture python stdout and stderr

[Paul M. <pau...@gm...>]

Glad I could give you some idea of a solution (even though I only have a passing familiarity with python).  Sometimes you just need to throw a few ideas at the wall, and see which ones stick.
This is where Brainstorming sessions (with no judgement, so even seemingly stupid ideas are fully considered) are really useful.

When you have a complete solution, I think anyone else who works with python and modules would benefit if you posted a link to a complete discussion/presentation (with code) of what you have come up with. So anyone else wrestling with this, can hopefully find that link with a search engine (instead of having to dig though a list discussion).

You should also let Xavier know if his proposed change to the module function in python might be helpful (look at his last post with this subject).


On 2022-03-23 08:25 AM, Jonathan Buck wrote:
> Hi Paul,
> 
> Thanks for the response!  I took a look at this and spoke with a colleague,
> but exit status is hard to grab from the module() method within python.
> The only way we were able to pull this off was to ignore the suggested
> python exec() statement found in the documentation and rather create our
> own popen() statement.  This allowed us to then check the status of that
> process.  What I don't like about this solution is that we're depending on
> the originating shell to have properly sourced the module environment, so
> that we can execute these 'os' commands.  Here's a snippet of the code my
> colleague came up with:
> 
> for mod in modules_to_test:    proc = subprocess.run(f"module use
> {mod_path};module load {mod}",            stdout=subprocess.PIPE,
> stderr=subprocess.PIPE, shell=True,
> universal_newlines=True)
>      print("", "########", f"Test: {mod}", "########", sep="\n")    #
> Just FYI to see the attributes of the object    # print(proc)
> print(f"stdout: {proc.stdout}")    print(f"stderr: {proc.stderr}")
> print(f"return code: {proc.returncode}")
> 
> We Found that all output goes to proc.stdout rather than any errors going
> to proc.stderr, regardless of a return code being 1 or 0.  The return code
> is the winner here. So, this works but I'm calling it a dirty solution
> because I think that once I automate this and try to execute remotely on
> our clusters, we'll lose the properly sourced environment.
> 
> On Tue, Mar 22, 2022 at 4:19 PM Paul Markfort <pau...@gm...> wrote:
> 
>> If you are just trying to, determine if there was an error thrown by the
>> module command,
>> Just test the exit status:
>>
>> EXIT STATUS
>>          The module command exits with 0 if its execution succeed.
>> Otherwise  1
>>          is returned.
>>
>>
>> Keep in mind that every call to the module command redirects sys.stdout so
>> that the output is evaluated by python (so you probably can't capture that).
>> And sys.stderr redirection by a module command might depend on what stderr
>> is connected to.
>>
>> Last - stderr and stdout are buffered IO, so you may also have to do a
>> flush before reading them.
>>
>>
>> I should preface the below with, "I haven't coded in python":
>>
>> Maybe you need to test your code with no modules involved first.
>> You might want to read this (seems like stringIO() may have some
>> limitations).
>>
>> https://stackoverflow.com/questions/1218933/can-i-redirect-the-stdout-into-some-sort-of-string-buffer
>>
>>
>>
>>
>>
>>
>>
>> On 2022-03-22 09:18 AM, Jonathan Buck wrote:
>>> Hi Jacques,
>>>
>>> Thanks for the quick reply!  I have successfully use the module() command
>>> to setup environments without issue.  What I'm hoping to do is capture
>> any
>>> stderr messages to a variable.  I have several use cases for this, most
>>> importantly I'm attempting to automate checks against our hundreds of
>>> modules to determine if any of them are broken when loaded.  I'm unable
>> to
>>> redirect stdout/stderr in a manner that I've used before which I believe
>> is
>>> due to the command being a subprocess.  I'm not a python guru, but the
>>> popen statement declares " stdout=subprocess.PIPE" which I thought was
>> the
>>> right way to get the stdout returned to the originating process.
>>>
>>> All that being said, here's my test script:
>>>
>>> import os
>>> import sys
>>> from io import StringIO
>>>
>>> # Setup variables to hold stdout/stderr
>>> result_stdout = StringIO()
>>> result_stderr = StringIO()
>>> sys.stdout = result_stdout
>>> sys.stderr = result_stderr
>>>
>>> # setup environment module
>>> exec(open('/p/app/Modules/4.7.1/init/python.py').read())
>>>
>>> # run
>>> print("----- TRY RUNNING MODULE COMMAND -----")
>>> module('--version')
>>> print("----- DONE RUNNING MODULE COMMAND -----")
>>> module_stdout = result_stdout.getvalue()
>>> module_stderr = result_stderr.getvalue()
>>>
>>> # reset stdout/stderr to defaults
>>> sys.stdout = sys.__stdout__
>>> sys.stderr = sys.__stderr__
>>>
>>> # print out what was captured to the variables
>>> print("===== TRY PRINTING STDOUT =====")
>>> sys.stdout.write(module_stdout)
>>> print("===== DONE PRINTING STDOUT =====")
>>>
>>> print("***** TRY PRINTING STDERR *****")
>>> sys.stderr.write(module_stderr)
>>> print("***** DONE PRINTING STDERR *****")
>>>
>>>
>>> As you can see, it's simple.  I make sure to reset stdout/stderr to
>>> defaults, however these are my results:
>>>
>>> ~> python3 module_test.py
>>> Modules Release 4.7.1 (2021-04-06)
>>> ===== TRY PRINTING JSON STDOUT =====
>>> ----- TRY RUNNING MODULE COMMAND -----
>>> ----- DONE RUNNING MODULE COMMAND -----
>>> ===== DONE PRINTING JSON STDOUT =====
>>> ***** TRY PRINTING JSON STDERR *****
>>> ***** DONE PRINTING JSON STDERR *****
>>> ~>
>>>
>>> This shows that my stdout redirect is capturing the 'print' statements
>>> for running the module command.  However the output for the module
>>> command prints at the very beginning and doesn't show up.  Any
>>> thoughts?  I found a fancy way to do the same thing in bash, and I
>>> proved that in bash the same output is going to stdout.
>>>
>>>
>>> Very Respectfully,
>>>
>>> Jonathan
>>>
>>>
>>> On Tue, Mar 22, 2022 at 8:06 AM Contact <dv...@cr...> wrote:
>>>
>>>> Hello,
>>>>
>>>> In my case I used modules from python with the following logic:
>>>>
>>>> import os
>>>> exec(open(/home/jrp/modules/init/python.py')).read())
>>>> module('avail')
>>>> module('load', 'gcc/7')
>>>> module('list')
>>>>
>>>>   From the above example,
>>>>
>>>>      - exec call will allow me to use modules from python itself and
>> define
>>>>      module function
>>>>      - Each module function invocation will be redirected to module to
>>>>      perform required action
>>>>
>>>> As reminder, the content of stdout is used by modules to setup all
>>>> required environment functions, functions/alias (if supported by current
>>>> shell), ...
>>>>
>>>> I hope it will help you.
>>>>
>>>> Jacques
>>>>
>>>>
>>>> Le 21/03/2022 à 20:09, Jonathan Buck a écrit :
>>>>
>>>> Hi,
>>>>
>>>> I'm attempting to capture and act on the stdout and stderr of the module
>>>> command.  The python implementation appears to be a subprocess for
>> which I
>>>> wasn't able to redirect with io.StringIO() as I have with other output
>>>> capture.
>>>>
>>>> I attempted to look through the code, which I admit I don't fully
>>>> understand and appears to have a one-liner style of executing the
>>>> subprocess.  I'm writing code that integrates with a production system,
>> so
>>>> I'm not able to modify the python.py code.
>>>>
>>>> Does anyone know how else this might be accomplished?  Or should I make
>>>> this a feature request?
>>>>
>>>> Thanks for your help,
>>>> Jonathan
>>>>
>>>>
>>>> _______________________________________________
>>>> Modules-interest mailing listModules-interest
>> @lists.sourceforge.nethttps://
>> lists.sourceforge.net/lists/listinfo/modules-interest
>>>>
>>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Modules-interest mailing list
>>> Mod...@li...
>>> https://lists.sourceforge.net/lists/listinfo/modules-interest
>>>
>>
>> --
>> --------------------------------------------------------
>> The views and opinions expressed above are strictly
>> those of the author(s).  The content of this message has
>> not been reviewed nor approved by any entity whatsoever.
>> --------------------------------------------------------
>> Paul FM         Info: http://paulfm.com/~paulfm/
>> --------------------------------------------------------
>>
>>
>> _______________________________________________
>> Modules-interest mailing list
>> Mod...@li...
>> https://lists.sourceforge.net/lists/listinfo/modules-interest
>>
> 
> 
> 
> _______________________________________________
> Modules-interest mailing list
> Mod...@li...
> https://lists.sourceforge.net/lists/listinfo/modules-interest
> 

-- 
--------------------------------------------------------
The views and opinions expressed above are strictly
those of the author(s).  The content of this message has
not been reviewed nor approved by any entity whatsoever.
--------------------------------------------------------
Paul FM         Info: http://paulfm.com/~paulfm/
--------------------------------------------------------

Re: [Modules] Capture python stdout and stderr

[Jonathan B. <jon...@gm...>]

Hi Paul,

Thanks for the response!  I took a look at this and spoke with a colleague,
but exit status is hard to grab from the module() method within python.
The only way we were able to pull this off was to ignore the suggested
python exec() statement found in the documentation and rather create our
own popen() statement.  This allowed us to then check the status of that
process.  What I don't like about this solution is that we're depending on
the originating shell to have properly sourced the module environment, so
that we can execute these 'os' commands.  Here's a snippet of the code my
colleague came up with:

for mod in modules_to_test:    proc = subprocess.run(f"module use
{mod_path};module load {mod}",            stdout=subprocess.PIPE,
stderr=subprocess.PIPE, shell=True,
universal_newlines=True)
    print("", "########", f"Test: {mod}", "########", sep="\n")    #
Just FYI to see the attributes of the object    # print(proc)
print(f"stdout: {proc.stdout}")    print(f"stderr: {proc.stderr}")
print(f"return code: {proc.returncode}")

We Found that all output goes to proc.stdout rather than any errors going
to proc.stderr, regardless of a return code being 1 or 0.  The return code
is the winner here. So, this works but I'm calling it a dirty solution
because I think that once I automate this and try to execute remotely on
our clusters, we'll lose the properly sourced environment.

On Tue, Mar 22, 2022 at 4:19 PM Paul Markfort <pau...@gm...> wrote:

> If you are just trying to, determine if there was an error thrown by the
> module command,
> Just test the exit status:
>
> EXIT STATUS
>         The module command exits with 0 if its execution succeed.
> Otherwise  1
>         is returned.
>
>
> Keep in mind that every call to the module command redirects sys.stdout so
> that the output is evaluated by python (so you probably can't capture that).
> And sys.stderr redirection by a module command might depend on what stderr
> is connected to.
>
> Last - stderr and stdout are buffered IO, so you may also have to do a
> flush before reading them.
>
>
> I should preface the below with, "I haven't coded in python":
>
> Maybe you need to test your code with no modules involved first.
> You might want to read this (seems like stringIO() may have some
> limitations).
>
> https://stackoverflow.com/questions/1218933/can-i-redirect-the-stdout-into-some-sort-of-string-buffer
>
>
>
>
>
>
>
> On 2022-03-22 09:18 AM, Jonathan Buck wrote:
> > Hi Jacques,
> >
> > Thanks for the quick reply!  I have successfully use the module() command
> > to setup environments without issue.  What I'm hoping to do is capture
> any
> > stderr messages to a variable.  I have several use cases for this, most
> > importantly I'm attempting to automate checks against our hundreds of
> > modules to determine if any of them are broken when loaded.  I'm unable
> to
> > redirect stdout/stderr in a manner that I've used before which I believe
> is
> > due to the command being a subprocess.  I'm not a python guru, but the
> > popen statement declares " stdout=subprocess.PIPE" which I thought was
> the
> > right way to get the stdout returned to the originating process.
> >
> > All that being said, here's my test script:
> >
> > import os
> > import sys
> > from io import StringIO
> >
> > # Setup variables to hold stdout/stderr
> > result_stdout = StringIO()
> > result_stderr = StringIO()
> > sys.stdout = result_stdout
> > sys.stderr = result_stderr
> >
> > # setup environment module
> > exec(open('/p/app/Modules/4.7.1/init/python.py').read())
> >
> > # run
> > print("----- TRY RUNNING MODULE COMMAND -----")
> > module('--version')
> > print("----- DONE RUNNING MODULE COMMAND -----")
> > module_stdout = result_stdout.getvalue()
> > module_stderr = result_stderr.getvalue()
> >
> > # reset stdout/stderr to defaults
> > sys.stdout = sys.__stdout__
> > sys.stderr = sys.__stderr__
> >
> > # print out what was captured to the variables
> > print("===== TRY PRINTING STDOUT =====")
> > sys.stdout.write(module_stdout)
> > print("===== DONE PRINTING STDOUT =====")
> >
> > print("***** TRY PRINTING STDERR *****")
> > sys.stderr.write(module_stderr)
> > print("***** DONE PRINTING STDERR *****")
> >
> >
> > As you can see, it's simple.  I make sure to reset stdout/stderr to
> > defaults, however these are my results:
> >
> > ~> python3 module_test.py
> > Modules Release 4.7.1 (2021-04-06)
> > ===== TRY PRINTING JSON STDOUT =====
> > ----- TRY RUNNING MODULE COMMAND -----
> > ----- DONE RUNNING MODULE COMMAND -----
> > ===== DONE PRINTING JSON STDOUT =====
> > ***** TRY PRINTING JSON STDERR *****
> > ***** DONE PRINTING JSON STDERR *****
> > ~>
> >
> > This shows that my stdout redirect is capturing the 'print' statements
> > for running the module command.  However the output for the module
> > command prints at the very beginning and doesn't show up.  Any
> > thoughts?  I found a fancy way to do the same thing in bash, and I
> > proved that in bash the same output is going to stdout.
> >
> >
> > Very Respectfully,
> >
> > Jonathan
> >
> >
> > On Tue, Mar 22, 2022 at 8:06 AM Contact <dv...@cr...> wrote:
> >
> >> Hello,
> >>
> >> In my case I used modules from python with the following logic:
> >>
> >> import os
> >> exec(open(/home/jrp/modules/init/python.py')).read())
> >> module('avail')
> >> module('load', 'gcc/7')
> >> module('list')
> >>
> >>  From the above example,
> >>
> >>     - exec call will allow me to use modules from python itself and
> define
> >>     module function
> >>     - Each module function invocation will be redirected to module to
> >>     perform required action
> >>
> >> As reminder, the content of stdout is used by modules to setup all
> >> required environment functions, functions/alias (if supported by current
> >> shell), ...
> >>
> >> I hope it will help you.
> >>
> >> Jacques
> >>
> >>
> >> Le 21/03/2022 à 20:09, Jonathan Buck a écrit :
> >>
> >> Hi,
> >>
> >> I'm attempting to capture and act on the stdout and stderr of the module
> >> command.  The python implementation appears to be a subprocess for
> which I
> >> wasn't able to redirect with io.StringIO() as I have with other output
> >> capture.
> >>
> >> I attempted to look through the code, which I admit I don't fully
> >> understand and appears to have a one-liner style of executing the
> >> subprocess.  I'm writing code that integrates with a production system,
> so
> >> I'm not able to modify the python.py code.
> >>
> >> Does anyone know how else this might be accomplished?  Or should I make
> >> this a feature request?
> >>
> >> Thanks for your help,
> >> Jonathan
> >>
> >>
> >> _______________________________________________
> >> Modules-interest mailing listModules-interest
> @lists.sourceforge.nethttps://
> lists.sourceforge.net/lists/listinfo/modules-interest
> >>
> >>
> >
> >
> >
> > _______________________________________________
> > Modules-interest mailing list
> > Mod...@li...
> > https://lists.sourceforge.net/lists/listinfo/modules-interest
> >
>
> --
> --------------------------------------------------------
> The views and opinions expressed above are strictly
> those of the author(s).  The content of this message has
> not been reviewed nor approved by any entity whatsoever.
> --------------------------------------------------------
> Paul FM         Info: http://paulfm.com/~paulfm/
> --------------------------------------------------------
>
>
> _______________________________________________
> Modules-interest mailing list
> Mod...@li...
> https://lists.sourceforge.net/lists/listinfo/modules-interest
>

[Modules] Lmod Tcl modulefile support

[Xavier D. <xav...@gm...>]

Hi All,

With upcoming v5.1 Modules will support Tcl modulefiles written for  Lmod,
the alternative *module* implementation developed in Lua. Such modulefiles
will be safely evaluated by Modules without raising error.

Support has been added for the following modulefile commands introduced by
Lmod: depends-on, prereq-any, always-load, module load-any, pushenv,
require-fullname and family.

A "Compatibility with Lmod Tcl modulefile" section [1] has been added into
the modulefile man page to describe the differences existing between the
two implementations.

Regards,
Xavier

[1]
https://modules.readthedocs.io/en/latest/modulefile.html#compatibility-with-lmod-tcl-modulefile-1

Re: [Modules] Does a linter for module files exist?

[Xavier D. <xav...@gm...>]

The "source syntaxbuild.tcl" + "buildFile mysyntaxdb.tcl" approach
described in the Nagelfar documentation does not seem to work when called
from a modulefile evaluation context. I have seen that there are also
".syntax" files : maybe that is a simpler approach to describe in such
files the additionnal procedures and variables brought by the modulefile
interpreter. But I do not know if global variables are supported in these
.syntax file.

Regards,
Xavier

Le lun. 21 mars 2022 à 21:35, Erik Parawell <eri...@si...> a
écrit :

> Thanks Xavier for the detailed answer!
>
> I’ll check out that formatting script and give it a test.
>
> In regards to Nagelfar, I’m looking into the syntaxbuild.tcl and it looks
> like it is partially hardcoded and then tries to extract things from the
> interpreter like subcommands.
> http://nagelfar.sourceforge.net/syntaxdatabases.html#creating-syntax-databases
>
> Do you know how to can run tcl scripts from the modulefile interpreter?
>
> On Mar 19, 2022, at 5:33 AM, Xavier Delaruelle <
> xav...@gm...> wrote:
>
>
> Le ven. 18 mars 2022 à 22:01, Erik Parawell <eri...@si...> a
> écrit :
>
>> Hi,
>>
>> I’m managing a lot of module files and I was wondering if there is a
>> linter for modules files or maybe some formatter (espeically for
>> indentation). I have seen that nagelfar132 is a good linter for TCL code
>> and that you can create a new syntax database using a provided scrip, but I
>> can’t figure out if I can even run that script with the module interpreter.
>>
>> Suggestions are welcome!
>>
>> Regards,
>> Erik Parawell
>>
>
> Hi Erik,
>
> As far as I know there is no linter existing for modulefiles. Best thing
> available at the moment is to use "module test" against the modulefiles to
> see if there are syntax errors.
>
> Using Nagelfar for linting seems quite interesting. Need to figure out how
> a new syntax db could be generated with all Tcl commands and variables
> added by the modulefile interpreter.
>
> Regarding code formatter, it seems there are not a lot of solution
> available out there. I have found (but not tested) this script proposed by
> the Tcl community:
> https://wiki.tcl-lang.org/page/Reformatting+Tcl+code+indentation
>
> Regards,
> Xavier
> _______________________________________________
> Modules-interest mailing list
> Mod...@li...
> https://lists.sourceforge.net/lists/listinfo/modules-interest
>
>
> _______________________________________________
> Modules-interest mailing list
> Mod...@li...
> https://lists.sourceforge.net/lists/listinfo/modules-interest
>

Re: [Modules] Capture python stdout and stderr

[Xavier D. <xav...@gm...>]

Hi Jonathan,

The current definition of the module function in python is:

$ $MODULES_CMD python autoinit
import os
import re, subprocess
def module(*arguments):
   ns = {}
   exec(subprocess.Popen(['/usr/bin/tclsh',
'/usr/share/Modules/libexec/modulecmd.tcl', 'python'] + list(arguments),
stdout=subprocess.PIPE).communicate()[0], ns)
   if '_mlstatus' in ns:
      _mlstatus = ns['_mlstatus']
   else:
      _mlstatus = True
   return _mlstatus
...

The stderr channel of the subprocess.Popen call is not defined, which means
that the content is sent directly to the stderr of the parent process. I
suppose that this is why you are not able to trap it.

Maybe the module function definition could be updated to add a
"stderr=susprocess.PIPE" argument to the subprocess.Popen call and then
print this content to sys.stderr. With this catch-then-release approach you
will be able to trap the content with your test code. I need to check if
there are side effects to such change.

import os, sys
import re, subprocess
from __future__ import print_function
def module(*arguments):
   ns = {}
   out, err = subprocess.Popen(['/usr/bin/tclsh',
'/usr/share/Modules/libexec/modulecmd.tcl', 'python'] + list(arguments),
stdout=subprocess.PIPE, stderr=susprocess.PIPE).communicate()
   exec(out, ns)
   if '_mlstatus' in ns:
      _mlstatus = ns['_mlstatus']
   else:
      _mlstatus = True
   print(err.decode(), file=sys.stderr)
   return _mlstatus

Regards,
Xavier


Le mar. 22 mars 2022 à 15:19, Jonathan Buck <jon...@gm...> a
écrit :

> Hi Jacques,
>
> Thanks for the quick reply!  I have successfully use the module() command
> to setup environments without issue.  What I'm hoping to do is capture any
> stderr messages to a variable.  I have several use cases for this, most
> importantly I'm attempting to automate checks against our hundreds of
> modules to determine if any of them are broken when loaded.  I'm unable to
> redirect stdout/stderr in a manner that I've used before which I believe is
> due to the command being a subprocess.  I'm not a python guru, but the
> popen statement declares " stdout=subprocess.PIPE" which I thought was the
> right way to get the stdout returned to the originating process.
>
> All that being said, here's my test script:
>
> import os
> import sys
> from io import StringIO
>
> # Setup variables to hold stdout/stderr
> result_stdout = StringIO()
> result_stderr = StringIO()
> sys.stdout = result_stdout
> sys.stderr = result_stderr
>
> # setup environment module
> exec(open('/p/app/Modules/4.7.1/init/python.py').read())
>
> # run
> print("----- TRY RUNNING MODULE COMMAND -----")
> module('--version')
> print("----- DONE RUNNING MODULE COMMAND -----")
> module_stdout = result_stdout.getvalue()
> module_stderr = result_stderr.getvalue()
>
> # reset stdout/stderr to defaults
> sys.stdout = sys.__stdout__
> sys.stderr = sys.__stderr__
>
> # print out what was captured to the variables
> print("===== TRY PRINTING STDOUT =====")
> sys.stdout.write(module_stdout)
> print("===== DONE PRINTING STDOUT =====")
>
> print("***** TRY PRINTING STDERR *****")
> sys.stderr.write(module_stderr)
> print("***** DONE PRINTING STDERR *****")
>
>
> As you can see, it's simple.  I make sure to reset stdout/stderr to defaults, however these are my results:
>
> ~> python3 module_test.py
> Modules Release 4.7.1 (2021-04-06)
> ===== TRY PRINTING JSON STDOUT =====
> ----- TRY RUNNING MODULE COMMAND -----
> ----- DONE RUNNING MODULE COMMAND -----
> ===== DONE PRINTING JSON STDOUT =====
> ***** TRY PRINTING JSON STDERR *****
> ***** DONE PRINTING JSON STDERR *****
> ~>
>
> This shows that my stdout redirect is capturing the 'print' statements for running the module command.  However the output for the module command prints at the very beginning and doesn't show up.  Any thoughts?  I found a fancy way to do the same thing in bash, and I proved that in bash the same output is going to stdout.
>
>
> Very Respectfully,
>
> Jonathan
>
>
> On Tue, Mar 22, 2022 at 8:06 AM Contact <dv...@cr...> wrote:
>
>> Hello,
>>
>> In my case I used modules from python with the following logic:
>>
>> import os
>> exec(open(/home/jrp/modules/init/python.py')).read())
>> module('avail')
>> module('load', 'gcc/7')
>> module('list')
>>
>> From the above example,
>>
>>    - exec call will allow me to use modules from python itself and
>>    define module function
>>    - Each module function invocation will be redirected to module to
>>    perform required action
>>
>> As reminder, the content of stdout is used by modules to setup all
>> required environment functions, functions/alias (if supported by current
>> shell), ...
>>
>> I hope it will help you.
>>
>> Jacques
>>
>>
>> Le 21/03/2022 à 20:09, Jonathan Buck a écrit :
>>
>> Hi,
>>
>> I'm attempting to capture and act on the stdout and stderr of the module
>> command.  The python implementation appears to be a subprocess for which I
>> wasn't able to redirect with io.StringIO() as I have with other output
>> capture.
>>
>> I attempted to look through the code, which I admit I don't fully
>> understand and appears to have a one-liner style of executing the
>> subprocess.  I'm writing code that integrates with a production system, so
>> I'm not able to modify the python.py code.
>>
>> Does anyone know how else this might be accomplished?  Or should I make
>> this a feature request?
>>
>> Thanks for your help,
>> Jonathan
>>
>>
>> _______________________________________________
>> Modules-interest mailing lis...@li...https://lists.sourceforge.net/lists/listinfo/modules-interest
>>
>> _______________________________________________
> Modules-interest mailing list
> Mod...@li...
> https://lists.sourceforge.net/lists/listinfo/modules-interest
>

Re: [Modules] Capture python stdout and stderr

[Paul M. <pau...@gm...>]

If you are just trying to, determine if there was an error thrown by the module command,
Just test the exit status:

EXIT STATUS
        The module command exits with 0 if its execution succeed.  Otherwise  1
        is returned.


Keep in mind that every call to the module command redirects sys.stdout so that the output is evaluated by python (so you probably can't capture that).
And sys.stderr redirection by a module command might depend on what stderr is connected to.

Last - stderr and stdout are buffered IO, so you may also have to do a flush before reading them.


I should preface the below with, "I haven't coded in python":

Maybe you need to test your code with no modules involved first.
You might want to read this (seems like stringIO() may have some limitations).
https://stackoverflow.com/questions/1218933/can-i-redirect-the-stdout-into-some-sort-of-string-buffer







On 2022-03-22 09:18 AM, Jonathan Buck wrote:
> Hi Jacques,
> 
> Thanks for the quick reply!  I have successfully use the module() command
> to setup environments without issue.  What I'm hoping to do is capture any
> stderr messages to a variable.  I have several use cases for this, most
> importantly I'm attempting to automate checks against our hundreds of
> modules to determine if any of them are broken when loaded.  I'm unable to
> redirect stdout/stderr in a manner that I've used before which I believe is
> due to the command being a subprocess.  I'm not a python guru, but the
> popen statement declares " stdout=subprocess.PIPE" which I thought was the
> right way to get the stdout returned to the originating process.
> 
> All that being said, here's my test script:
> 
> import os
> import sys
> from io import StringIO
> 
> # Setup variables to hold stdout/stderr
> result_stdout = StringIO()
> result_stderr = StringIO()
> sys.stdout = result_stdout
> sys.stderr = result_stderr
> 
> # setup environment module
> exec(open('/p/app/Modules/4.7.1/init/python.py').read())
> 
> # run
> print("----- TRY RUNNING MODULE COMMAND -----")
> module('--version')
> print("----- DONE RUNNING MODULE COMMAND -----")
> module_stdout = result_stdout.getvalue()
> module_stderr = result_stderr.getvalue()
> 
> # reset stdout/stderr to defaults
> sys.stdout = sys.__stdout__
> sys.stderr = sys.__stderr__
> 
> # print out what was captured to the variables
> print("===== TRY PRINTING STDOUT =====")
> sys.stdout.write(module_stdout)
> print("===== DONE PRINTING STDOUT =====")
> 
> print("***** TRY PRINTING STDERR *****")
> sys.stderr.write(module_stderr)
> print("***** DONE PRINTING STDERR *****")
> 
> 
> As you can see, it's simple.  I make sure to reset stdout/stderr to
> defaults, however these are my results:
> 
> ~> python3 module_test.py
> Modules Release 4.7.1 (2021-04-06)
> ===== TRY PRINTING JSON STDOUT =====
> ----- TRY RUNNING MODULE COMMAND -----
> ----- DONE RUNNING MODULE COMMAND -----
> ===== DONE PRINTING JSON STDOUT =====
> ***** TRY PRINTING JSON STDERR *****
> ***** DONE PRINTING JSON STDERR *****
> ~>
> 
> This shows that my stdout redirect is capturing the 'print' statements
> for running the module command.  However the output for the module
> command prints at the very beginning and doesn't show up.  Any
> thoughts?  I found a fancy way to do the same thing in bash, and I
> proved that in bash the same output is going to stdout.
> 
> 
> Very Respectfully,
> 
> Jonathan
> 
> 
> On Tue, Mar 22, 2022 at 8:06 AM Contact <dv...@cr...> wrote:
> 
>> Hello,
>>
>> In my case I used modules from python with the following logic:
>>
>> import os
>> exec(open(/home/jrp/modules/init/python.py')).read())
>> module('avail')
>> module('load', 'gcc/7')
>> module('list')
>>
>>  From the above example,
>>
>>     - exec call will allow me to use modules from python itself and define
>>     module function
>>     - Each module function invocation will be redirected to module to
>>     perform required action
>>
>> As reminder, the content of stdout is used by modules to setup all
>> required environment functions, functions/alias (if supported by current
>> shell), ...
>>
>> I hope it will help you.
>>
>> Jacques
>>
>>
>> Le 21/03/2022 à 20:09, Jonathan Buck a écrit :
>>
>> Hi,
>>
>> I'm attempting to capture and act on the stdout and stderr of the module
>> command.  The python implementation appears to be a subprocess for which I
>> wasn't able to redirect with io.StringIO() as I have with other output
>> capture.
>>
>> I attempted to look through the code, which I admit I don't fully
>> understand and appears to have a one-liner style of executing the
>> subprocess.  I'm writing code that integrates with a production system, so
>> I'm not able to modify the python.py code.
>>
>> Does anyone know how else this might be accomplished?  Or should I make
>> this a feature request?
>>
>> Thanks for your help,
>> Jonathan
>>
>>
>> _______________________________________________
>> Modules-interest mailing lis...@li...https://lists.sourceforge.net/lists/listinfo/modules-interest
>>
>>
> 
> 
> 
> _______________________________________________
> Modules-interest mailing list
> Mod...@li...
> https://lists.sourceforge.net/lists/listinfo/modules-interest
> 

-- 
--------------------------------------------------------
The views and opinions expressed above are strictly
those of the author(s).  The content of this message has
not been reviewed nor approved by any entity whatsoever.
--------------------------------------------------------
Paul FM         Info: http://paulfm.com/~paulfm/
--------------------------------------------------------

Re: [Modules] Capture python stdout and stderr

[Jonathan B. <jon...@gm...>]

Hi Jacques,

Thanks for the quick reply!  I have successfully use the module() command
to setup environments without issue.  What I'm hoping to do is capture any
stderr messages to a variable.  I have several use cases for this, most
importantly I'm attempting to automate checks against our hundreds of
modules to determine if any of them are broken when loaded.  I'm unable to
redirect stdout/stderr in a manner that I've used before which I believe is
due to the command being a subprocess.  I'm not a python guru, but the
popen statement declares " stdout=subprocess.PIPE" which I thought was the
right way to get the stdout returned to the originating process.

All that being said, here's my test script:

import os
import sys
from io import StringIO

# Setup variables to hold stdout/stderr
result_stdout = StringIO()
result_stderr = StringIO()
sys.stdout = result_stdout
sys.stderr = result_stderr

# setup environment module
exec(open('/p/app/Modules/4.7.1/init/python.py').read())

# run
print("----- TRY RUNNING MODULE COMMAND -----")
module('--version')
print("----- DONE RUNNING MODULE COMMAND -----")
module_stdout = result_stdout.getvalue()
module_stderr = result_stderr.getvalue()

# reset stdout/stderr to defaults
sys.stdout = sys.__stdout__
sys.stderr = sys.__stderr__

# print out what was captured to the variables
print("===== TRY PRINTING STDOUT =====")
sys.stdout.write(module_stdout)
print("===== DONE PRINTING STDOUT =====")

print("***** TRY PRINTING STDERR *****")
sys.stderr.write(module_stderr)
print("***** DONE PRINTING STDERR *****")


As you can see, it's simple.  I make sure to reset stdout/stderr to
defaults, however these are my results:

~> python3 module_test.py
Modules Release 4.7.1 (2021-04-06)
===== TRY PRINTING JSON STDOUT =====
----- TRY RUNNING MODULE COMMAND -----
----- DONE RUNNING MODULE COMMAND -----
===== DONE PRINTING JSON STDOUT =====
***** TRY PRINTING JSON STDERR *****
***** DONE PRINTING JSON STDERR *****
~>

This shows that my stdout redirect is capturing the 'print' statements
for running the module command.  However the output for the module
command prints at the very beginning and doesn't show up.  Any
thoughts?  I found a fancy way to do the same thing in bash, and I
proved that in bash the same output is going to stdout.


Very Respectfully,

Jonathan


On Tue, Mar 22, 2022 at 8:06 AM Contact <dv...@cr...> wrote:

> Hello,
>
> In my case I used modules from python with the following logic:
>
> import os
> exec(open(/home/jrp/modules/init/python.py')).read())
> module('avail')
> module('load', 'gcc/7')
> module('list')
>
> From the above example,
>
>    - exec call will allow me to use modules from python itself and define
>    module function
>    - Each module function invocation will be redirected to module to
>    perform required action
>
> As reminder, the content of stdout is used by modules to setup all
> required environment functions, functions/alias (if supported by current
> shell), ...
>
> I hope it will help you.
>
> Jacques
>
>
> Le 21/03/2022 à 20:09, Jonathan Buck a écrit :
>
> Hi,
>
> I'm attempting to capture and act on the stdout and stderr of the module
> command.  The python implementation appears to be a subprocess for which I
> wasn't able to redirect with io.StringIO() as I have with other output
> capture.
>
> I attempted to look through the code, which I admit I don't fully
> understand and appears to have a one-liner style of executing the
> subprocess.  I'm writing code that integrates with a production system, so
> I'm not able to modify the python.py code.
>
> Does anyone know how else this might be accomplished?  Or should I make
> this a feature request?
>
> Thanks for your help,
> Jonathan
>
>
> _______________________________________________
> Modules-interest mailing lis...@li...https://lists.sourceforge.net/lists/listinfo/modules-interest
>
>

Re: [Modules] Capture python stdout and stderr

[Contact <dv...@cr...>]

Hello,

In my case I used modules from python with the following logic:

import os
exec(open(/home/jrp/modules/init/python.py')).read())
module('avail')
module('load', 'gcc/7')
module('list')

 From the above example,

  * exec call will allow me to use modules from python itself and define
    module function
  * Each module function invocation will be redirected to module to
    perform required action

As reminder, the content of stdout is used by modules to setup all 
required environment functions, functions/alias (if supported by current 
shell), ...

I hope it will help you.

Jacques


Le 21/03/2022 à 20:09, Jonathan Buck a écrit :
> Hi,
>
> I'm attempting to capture and act on the stdout and stderr of the 
> module command.  The python implementation appears to be a subprocess 
> for which I wasn't able to redirect with io.StringIO() as I have with 
> other output capture.
>
> I attempted to look through the code, which I admit I don't fully 
> understand and appears to have a one-liner style of executing the 
> subprocess.  I'm writing code that integrates with a production 
> system, so I'm not able to modify the python.py code.
>
> Does anyone know how else this might be accomplished?  Or should I 
> make this a feature request?
>
> Thanks for your help,
> Jonathan
>
>
> _______________________________________________
> Modules-interest mailing list
> Mod...@li...
> https://lists.sourceforge.net/lists/listinfo/modules-interest

Re: [Modules] Capture python stdout and stderr

[Contact <dv...@cr...>]

Hello,

In my case I used modules from python with the following logic:

import os
exec(open(/home/jrp/modules/init/python.py')).read())
module('avail')
module('load', 'gcc/7')
module('list')

 From the above example,

  * exec call will allow me to use modules from python itself and define
    module function
  * Each module function invocation will be redirected to module to
    perform required action

As reminder, the content of stdout is used by modules to setup all 
required environment functions, functions/alias (if supported by current 
shell), ...

I hope it will help you.

Jacques


Le 21/03/2022 à 20:09, Jonathan Buck a écrit :
> Hi,
>
> I'm attempting to capture and act on the stdout and stderr of the 
> module command.  The python implementation appears to be a subprocess 
> for which I wasn't able to redirect with io.StringIO() as I have with 
> other output capture.
>
> I attempted to look through the code, which I admit I don't fully 
> understand and appears to have a one-liner style of executing the 
> subprocess.  I'm writing code that integrates with a production 
> system, so I'm not able to modify the python.py code.
>
> Does anyone know how else this might be accomplished?  Or should I 
> make this a feature request?
>
> Thanks for your help,
> Jonathan
>
>
> _______________________________________________
> Modules-interest mailing list
> Mod...@li...
> https://lists.sourceforge.net/lists/listinfo/modules-interest

Re: [Modules] Does a linter for module files exist?

[Erik P. <eri...@si...>]

Thanks Xavier for the detailed answer!

I’ll check out that formatting script and give it a test.

In regards to Nagelfar, I’m looking into the syntaxbuild.tcl and it looks like it is partially hardcoded and then tries to extract things from the interpreter like subcommands. http://nagelfar.sourceforge.net/syntaxdatabases.html#creating-syntax-databases <http://nagelfar.sourceforge.net/syntaxdatabases.html#creating-syntax-databases>

Do you know how to can run tcl scripts from the modulefile interpreter?

> On Mar 19, 2022, at 5:33 AM, Xavier Delaruelle <xav...@gm...> wrote:
> 
> 
> Le ven. 18 mars 2022 à 22:01, Erik Parawell <eri...@si... <mailto:eri...@si...>> a écrit :
> Hi,
> 
> I’m managing a lot of module files and I was wondering if there is a linter for modules files or maybe some formatter (espeically for indentation). I have seen that nagelfar132 is a good linter for TCL code and that you can create a new syntax database using a provided scrip, but I can’t figure out if I can even run that script with the module interpreter.
> 
> Suggestions are welcome!
> 
> Regards,
> Erik Parawell
> 
> Hi Erik,
> 
> As far as I know there is no linter existing for modulefiles. Best thing available at the moment is to use "module test" against the modulefiles to see if there are syntax errors.
> 
> Using Nagelfar for linting seems quite interesting. Need to figure out how a new syntax db could be generated with all Tcl commands and variables added by the modulefile interpreter.
> 
> Regarding code formatter, it seems there are not a lot of solution available out there. I have found (but not tested) this script proposed by the Tcl community:
> https://wiki.tcl-lang.org/page/Reformatting+Tcl+code+indentation <https://wiki.tcl-lang.org/page/Reformatting+Tcl+code+indentation>
> 
> Regards,
> Xavier
> _______________________________________________
> Modules-interest mailing list
> Mod...@li...
> https://lists.sourceforge.net/lists/listinfo/modules-interest

[Modules] Capture python stdout and stderr

[Jonathan B. <jon...@gm...>]

Hi,

I'm attempting to capture and act on the stdout and stderr of the module
command.  The python implementation appears to be a subprocess for which I
wasn't able to redirect with io.StringIO() as I have with other output
capture.

I attempted to look through the code, which I admit I don't fully
understand and appears to have a one-liner style of executing the
subprocess.  I'm writing code that integrates with a production system, so
I'm not able to modify the python.py code.

Does anyone know how else this might be accomplished?  Or should I make
this a feature request?

Thanks for your help,
Jonathan

Re: [Modules] Environment Modules for Multi-arch Environment

[Michael K. <mi...@do...>]

Hi everyone,

On 21.03.2022 07:29, Michael Klemm wrote:
 > On 19.03.2022 14:19, Xavier Delaruelle wrote:
 >> Virtual modules [1] seem a good solution for this situation. The idea
 >> is to dynamically define the available modulefiles based on the
 >> software directories found under the current $ARCH directory.
 >
 > That sounds like a good way to implement this.  Let me try this and see
 > if I still have some questions left, but it looks like the ideal
 > solution to my problem.

After I saw that Modules >4.7 has module-hide and module-forbid, I went 
this way now (after an update of the modules package):

------------ snip ------------
#%Module

set archdir /net/software/$env(ARCH)

set softdir /net/software
set modprefix $softdir/modules/modulefiles

foreach modfile [glob $modprefix/*/*.*] {
     regsub ***=/net/software/modules/modulefiles/ $modfile "" mod
     if {![file exists $archdir/$mod] && ![file exists $softdir/$mod]} {
         module-hide --soft $mod
         module-forbid $mod
     }
}
------------ snap ------------

Kind regards,
         -michael

Re: [Modules] Environment Modules for Multi-arch Environment

[Michael K. <mi...@do...>]

Hi Xavier,

On 19.03.2022 14:19, Xavier Delaruelle wrote:
> Virtual modules [1] seem a good solution for this situation. The idea is 
> to dynamically define the available modulefiles based on the software 
> directories found under the current $ARCH directory.

That sounds like a good way to implement this.  Let me try this and see 
if I still have some questions left, but it looks like the ideal 
solution to my problem.


> Instead of having a modulefile per software-version couple, you could 
> just have a single file that adapts its evaluation based on the name it 
> has been called (use [module-info name] in modulefile to fetch the name 
> of the module currently being evaluated).
[...]

Thanks for the detailed description.  It will take a bit to implement 
it, but I will certainly try it out in the next couple of days.

Kind regards,
         -michael

Re: [Modules] Environment Modules for Multi-arch Environment

[Xavier D. <xav...@gm...>]

Hi Michael and Nick,

Virtual modules [1] seem a good solution for this situation. The idea is to
dynamically define the available modulefiles based on the software
directories found under the current $ARCH directory.

Instead of having a modulefile per software-version couple, you could just
have a single file that adapts its evaluation based on the name it has been
called (use [module-info name] in modulefile to fetch the name of the
module currently being evaluated).

$ tree software/modules/modulefiles/
software/modules/modulefiles/
├── clang
└── gcc

Then create a specific modulepath directory containing only a .modulerc
file containing the code that evaluate the available software installation
for current ARCH and then define virtual modules with the module-virtual
modulefile command:

$ tree -a software/modules/modulepath/
software/modules/modulepath/
└── .modulerc

0 directories, 1 file
$ cat software/modules/modulepath/.modulerc
#%Module

set archdir ../../$env(ARCH)/
foreach softdir [glob ${archdir}*/*] {
    set mod [string range $softdir [string length $archdir] end]
    set modname [file dirname $mod]
    module-virtual $mod ../modulefiles/$modname
}

Once files have been set, just enable the modulepath and set the ARCH
variable:

$ tree software/{aarch64,x86_64}
software/aarch64
├── clang
│   └── 12.0.0
└── gcc
    └── 10.1.0
software/x86_64
├── clang
│   └── 13.0.0
└── gcc
    └── 9.2.0

8 directories, 0 files
$ module use software/modules/modulepath
$ export ARCH=x86_64
$ module avail
---- /path/to/software/modules/modulepath ----
clang/13.0.0  gcc/9.2.0

$ export ARCH=aarch64
$ module avail
---- /path/to/software/modules/modulepath ----
clang/12.0.0  gcc/10.1.0

Regards,
Xavier

[1]
https://modules.readthedocs.io/en/latest/modulefile.html#mfcmd-module-virtual

Le sam. 19 mars 2022 à 10:21, Nick Papior <nic...@gm...> a écrit :

> One solution would be to duplicate the module files for each arch, then a
> presetup determines the path of modules to be enabled and you won't have to
> make specific checks in module files. Of course this will duplicate module
> files, but it will also allow easier debugging and changing global
> variables per arch. I find this more convenient.
>
> On Sat, 19 Mar 2022, 08:44 Michael Klemm, <mi...@do...> wrote:
>
>> Folks,
>>
>> I have the following setup:
>>
>> /net/software/{x86_64,aarch64,armv7l,riscv} contains all software on
>> shared storage.  /net/software/modules/modulefiles contains the
>> corresponding module files for these software packages, e.g.,
>> clang/13.0.0, gcc/9.2.0, etc.  The module files use the $ARCH
>> environment variable to setup the environment for the software.
>>
>> Since I do not have all software packages compiled for all systems, I
>> was wondering what would be the best way to avoid loading (or even
>> better hide) modules that do not have the software package for a
>> particular architecture?
>>
>> I have come up with either checking in the module file, if the software
>> exists and if not throw some sort of error.  I could also have
>> $ARCH-specific module locations and use "module use" on these locations,
>> depending on the machine I'm on.  But is there a better way to automate
>> all this w/o too much duplication of module files or code in the module
>> files?
>>
>> I'm using Modules Release 4.4.1 (2020-01-03), all systems are Ubuntu
>> systems.
>>
>> Kind regards,
>>          -michael
>>
>> _______________________________________________
>> Modules-interest mailing list
>> Mod...@li...
>> https://lists.sourceforge.net/lists/listinfo/modules-interest
>>
> _______________________________________________
> Modules-interest mailing list
> Mod...@li...
> https://lists.sourceforge.net/lists/listinfo/modules-interest
>

Re: [Modules] Does a linter for module files exist?

[Xavier D. <xav...@gm...>]

Le ven. 18 mars 2022 à 22:01, Erik Parawell <eri...@si...> a
écrit :

> Hi,
>
> I’m managing a lot of module files and I was wondering if there is a
> linter for modules files or maybe some formatter (espeically for
> indentation). I have seen that nagelfar132 is a good linter for TCL code
> and that you can create a new syntax database using a provided scrip, but I
> can’t figure out if I can even run that script with the module interpreter.
>
> Suggestions are welcome!
>
> Regards,
> Erik Parawell
>

Hi Erik,

As far as I know there is no linter existing for modulefiles. Best thing
available at the moment is to use "module test" against the modulefiles to
see if there are syntax errors.

Using Nagelfar for linting seems quite interesting. Need to figure out how
a new syntax db could be generated with all Tcl commands and variables
added by the modulefile interpreter.

Regarding code formatter, it seems there are not a lot of solution
available out there. I have found (but not tested) this script proposed by
the Tcl community:
https://wiki.tcl-lang.org/page/Reformatting+Tcl+code+indentation

Regards,
Xavier

Re: [Modules] Environment Modules for Multi-arch Environment

[Nick P. <nic...@gm...>]

One solution would be to duplicate the module files for each arch, then a
presetup determines the path of modules to be enabled and you won't have to
make specific checks in module files. Of course this will duplicate module
files, but it will also allow easier debugging and changing global
variables per arch. I find this more convenient.

On Sat, 19 Mar 2022, 08:44 Michael Klemm, <mi...@do...> wrote:

> Folks,
>
> I have the following setup:
>
> /net/software/{x86_64,aarch64,armv7l,riscv} contains all software on
> shared storage.  /net/software/modules/modulefiles contains the
> corresponding module files for these software packages, e.g.,
> clang/13.0.0, gcc/9.2.0, etc.  The module files use the $ARCH
> environment variable to setup the environment for the software.
>
> Since I do not have all software packages compiled for all systems, I
> was wondering what would be the best way to avoid loading (or even
> better hide) modules that do not have the software package for a
> particular architecture?
>
> I have come up with either checking in the module file, if the software
> exists and if not throw some sort of error.  I could also have
> $ARCH-specific module locations and use "module use" on these locations,
> depending on the machine I'm on.  But is there a better way to automate
> all this w/o too much duplication of module files or code in the module
> files?
>
> I'm using Modules Release 4.4.1 (2020-01-03), all systems are Ubuntu
> systems.
>
> Kind regards,
>          -michael
>
> _______________________________________________
> Modules-interest mailing list
> Mod...@li...
> https://lists.sourceforge.net/lists/listinfo/modules-interest
>

[Modules] Environment Modules for Multi-arch Environment

[Michael K. <mi...@do...>]

Folks,

I have the following setup:

/net/software/{x86_64,aarch64,armv7l,riscv} contains all software on 
shared storage.  /net/software/modules/modulefiles contains the 
corresponding module files for these software packages, e.g., 
clang/13.0.0, gcc/9.2.0, etc.  The module files use the $ARCH 
environment variable to setup the environment for the software.

Since I do not have all software packages compiled for all systems, I 
was wondering what would be the best way to avoid loading (or even 
better hide) modules that do not have the software package for a 
particular architecture?

I have come up with either checking in the module file, if the software 
exists and if not throw some sort of error.  I could also have 
$ARCH-specific module locations and use "module use" on these locations, 
depending on the machine I'm on.  But is there a better way to automate 
all this w/o too much duplication of module files or code in the module 
files?

I'm using Modules Release 4.4.1 (2020-01-03), all systems are Ubuntu 
systems.

Kind regards,
         -michael