Re: [Pythondialog-users] Example usage of Programbox

[Florent R. <f.r...@fr...>]

Hi,

Paraic OCeallaigh <par...@gm...> wrote:

> Thanks for a great product! I have several menu systems on many Linux
> servers using it in Production.

That's nice to hear, thank you.

> My issue:
> I am trying to use programbox to show a running display of a health check
> on a Linux box.
> I would run several os.system() commands such as service xxxx status and df
> and top etc whcih would append their stdout to a file in /tmp.
> I would like to see these commands send output to a programbox (or
> progressbox) either via some kind of pipe or tailing the file in /tmp.

First advice: don't use os.system() unless you have a compelling reason.
It does too many undesirable things like arguments splitting around
spaces and parameter expansion, that are completely unnecessary when
using Python (we have lists/sequences, no need to special-case the space
character, the ";", etc.) and often open your code to security problems,
unless you are very careful (using shlex.quote() or similar). I
recommend you to replace such calls with:
  - the glob module (glob.glob() or glon.iglob() typically) if you need
    to expand shell-style patterns;
  - the subprocess module for most cases where you need to run a
    separate program, capture its stdout/stderr output or feed him data
    via its stdin;
  - for complex cases where the subprocess module doesn't offer what you
    need, you can do the work by hand with os.fork() and os.execvp() (or
    os.execvpe() or... there are a few similar functions in the 'exec'
    family) + possibly os.dup2() to redirect stdin/out/err to a file or
    pipe, os.devnull/subprocess.DEVNULL, etc.). You can read
    Dialog._call_program() in dialog.py for an example of this, actually
    written before the subprocess module was born.

Using os.system(), very dangerous things can happen if your program
reads an argument such as "; rm -rf /;" from an untrusted source and
includes it in the command line. Variable/parameter expansions also
represent a high risk (for instance, I read that the Steam installer [a
shell script] ran a command equivalent to 'rm -rf /*' on customers'
systems because of some stupid "rm -rf "$VARIABLE/*" where $VARIABLE
happened to have an empty expansion. *** Don't use os.system()***. Write
convenience wrappers around 'subprocess' if you need, but don't use
os.system(). You will even gain in efficiency by not running unneeded
shell processes (in case you didn't realize, os.system() runs the
command via a shell, which is quite unneccessary in general).

************************************************************************
To address your question more specifically, I would need to know the
versions of Python and pythondialog you want this to work with, and how
precisely you want the output of the commands to be gathered and
displayed.
************************************************************************

First, if you want to run one programbox or progressbox per external
program (I suspect this is not what you want, but it is the simplest
approach, so let's start with it), the code running "find /usr/bin" in
demo.py (MyApp.programbox_demo()) is basically what you need, except you
can maybe simplify the /dev/null handling if you know the precise Python
versions you are targetting. Probably you'll want to use
'stderr=subprocess.STDOUT' in the subprocess call in order to see the
stderr output of external processes instead of it being thrown away.

OTOH, if you want to have one programbox or progressbox widget
continuously display the output of several commands, the best way would
probably be to:
  - create a pipe(7) whose reading end will be used by dialog for the
    whole widget life time, and whose writing end will receive the
    output of the programs, possibly mixed with output from your Python
    script;
  - start a child process (let's call it C1) with os.fork() that will
    inherit the file descriptor[1] corresponding to the writing end of
    the pipe, and use it to transmit the data you want to be displayed
    in the programbox or progressbox widget; each of the external
    programs will be started as a child process of C1, and C1 should
    wait(2) for the completion of every one of them;
  - in the main process, run d.programbox() or d.progressbox(), where d
    is your Dialog instance. This will last as long as there is at least
    one writer that hasn't closed the writing end of the pipe (a file
    descriptor can be duplicated with dup(2) [os.dup() in Python] and
    friends, as well as inherited by child processes upon fork(2)
    [os.fork() in Python]; each of these operations increases the
    "reference count" of the fd, which determines when the process
    reading from the pipe will see EOF instead of blocking in the
    read(2) system call, because this is when the "reference count" of
    the writing end of the pipe drops to 0 and all data previously in
    the pipe has been read that the reader process is told "there is no
    more data, this is over" [EOF]). Then your main process can wait for
    the child process C1 to exit.

[1] An integer, i.e., low-level (OS-level) "file", different from a
    Python file object. It is often a good idea to close the file
    descriptors you don't need in the child, e.g., the reading end of
    the pipe in C1. Sometimes, it is even necessary, otherwise some
    operation may block (writing end of the pipe in the main process,
    that would make the dialog reader process believe, if the fd were
    not closed, that there is still the possibility of data being sent
    through the pipe). Closing a file descriptor in a child does not
    close it for the parent; it just closes a duplicate fd, decrementing
    its "reference count".

This is exactly what MyApp.progressboxoid() in demo.py does, except that
instead of spawning child processes with their stdout and stderr file
descriptors assigned to the writing end of the pipe, it creates a Python
file object from the file descriptor corresponding to the writing end of
the pipe and writes to it as with any file object. This is the following
part of MyApp.progressboxoid():

   # Python file objects are easier to use than file descriptors.
   # For a start, you don't have to check the number of bytes
   # actually written every time...
   # "buffering = 1" means wfile is going to be line-buffered
   with os.fdopen(write_fd, mode="w", buffering=1) as wfile:
       for line in text.split('\n'):
           wfile.write(line + '\n')
           time.sleep(0.02 if params["fast_mode"] else 1.2)

(os.fdopen() can be replaced with open() in not-too-old Python versions,
but beware, arguments may vary slightly)

In order to start a child process from C1, running an external program
with its stdout and stderr redirected to the writing end of the pipe,
you could use something like:

  class YourException(Exception): # Name it appropriately (UnableToRunBlaBla...)
      pass


  program = "systemctl"
  args = [program, "status", "foobar.service"]
  try:
      with subprocess.Popen(args, stdin=subprocess.DEVNULL, stdout=write_fd,
                            stderr=subprocess.STDOUT,
                            universal_newlines=True) as proc:
          out = proc.stdout.read()
  except OSError as e:
      raise YourException(
          "unable to find or execute {0!r}".format(program)) from e

where 'write_fd' is the file descriptor corresponding to the writing
end of the pipe, cf. MyApp.progressboxoid().

Then you can inspect proc.returncode to determine the exit status of the
systemctl child process (if proc.returncode >= 0), or the signal that
killed him, if any (this is if proc.returncode < 0; the signal number is
-proc.returncode in this case).

There are very slightly shorter alternatives to subprocess.Popen() in
the subprocess module that are convenient in simple cases (run() in
Python 3.5 and later; otherwise, call(), check_output(), etc. but these
seem to be deprecated in Python 3.5). Anyway, subprocess.Popen() is the
most powerful one used as a basis for the others, it is not even
difficult to use, and not deprecated either.

I suggest you study MyApp.progressboxoid() and tell me precisely which
lines you don't understand. This may seem a bit complex at first, but it
is robust and uses resources efficiently. If you use pipes to transfer
the output of external programs to dialog, this output won't accumulate
in memory and your processes can run for years producing huge amounts of
data without any problem. This is essentially using the basic mechanisms
provided for this purpose by the OS (i.e., fork(2), exec(3) and
pipe(2)/pipe(7)). It is likely that the external programs you'll want to
run will differ very little if at all in the way they need to be
handled. Thus, you'll probably be able to use a common function or
method to handle all of them (this is essentially what pythondialog does
in dialog.py for the myriad of possible dialog commands, otherwise it
would be totally unmaintainable).

Note: you mentioned 'top'; in its default mode of operation, its output
      is probably unsuitable for sending to a pipe (except if it behaves
      differently when its stdout is not a terminal, this is possible).
      Probably something like "batch mode" (top's -b option) would be
      appropriate.

If you've followed this advice and still didn't manage to get something
working, post a minimal example of what you have tried.

> Would you be able to guide me on how best to do that and some sample code
> if possible? I looked at the demo.py and couldn't quite figure out the code
> to do it there. My python is still beginner-ish but very keen to learn.
>
> Any help appreciated

HTH :-)

-- 
Florent