From: Florent R. <f.r...@fr...> - 2015-11-26 11:58:00
|
Hi, Paraic OCeallaigh <par...@gm...> wrote: > Thanks for a great product! I have several menu systems on many Linux > servers using it in Production. That's nice to hear, thank you. > My issue: > I am trying to use programbox to show a running display of a health check > on a Linux box. > I would run several os.system() commands such as service xxxx status and df > and top etc whcih would append their stdout to a file in /tmp. > I would like to see these commands send output to a programbox (or > progressbox) either via some kind of pipe or tailing the file in /tmp. First advice: don't use os.system() unless you have a compelling reason. It does too many undesirable things like arguments splitting around spaces and parameter expansion, that are completely unnecessary when using Python (we have lists/sequences, no need to special-case the space character, the ";", etc.) and often open your code to security problems, unless you are very careful (using shlex.quote() or similar). I recommend you to replace such calls with: - the glob module (glob.glob() or glon.iglob() typically) if you need to expand shell-style patterns; - the subprocess module for most cases where you need to run a separate program, capture its stdout/stderr output or feed him data via its stdin; - for complex cases where the subprocess module doesn't offer what you need, you can do the work by hand with os.fork() and os.execvp() (or os.execvpe() or... there are a few similar functions in the 'exec' family) + possibly os.dup2() to redirect stdin/out/err to a file or pipe, os.devnull/subprocess.DEVNULL, etc.). You can read Dialog._call_program() in dialog.py for an example of this, actually written before the subprocess module was born. Using os.system(), very dangerous things can happen if your program reads an argument such as "; rm -rf /;" from an untrusted source and includes it in the command line. Variable/parameter expansions also represent a high risk (for instance, I read that the Steam installer [a shell script] ran a command equivalent to 'rm -rf /*' on customers' systems because of some stupid "rm -rf "$VARIABLE/*" where $VARIABLE happened to have an empty expansion. *** Don't use os.system()***. Write convenience wrappers around 'subprocess' if you need, but don't use os.system(). You will even gain in efficiency by not running unneeded shell processes (in case you didn't realize, os.system() runs the command via a shell, which is quite unneccessary in general). ************************************************************************ To address your question more specifically, I would need to know the versions of Python and pythondialog you want this to work with, and how precisely you want the output of the commands to be gathered and displayed. ************************************************************************ First, if you want to run one programbox or progressbox per external program (I suspect this is not what you want, but it is the simplest approach, so let's start with it), the code running "find /usr/bin" in demo.py (MyApp.programbox_demo()) is basically what you need, except you can maybe simplify the /dev/null handling if you know the precise Python versions you are targetting. Probably you'll want to use 'stderr=subprocess.STDOUT' in the subprocess call in order to see the stderr output of external processes instead of it being thrown away. OTOH, if you want to have one programbox or progressbox widget continuously display the output of several commands, the best way would probably be to: - create a pipe(7) whose reading end will be used by dialog for the whole widget life time, and whose writing end will receive the output of the programs, possibly mixed with output from your Python script; - start a child process (let's call it C1) with os.fork() that will inherit the file descriptor[1] corresponding to the writing end of the pipe, and use it to transmit the data you want to be displayed in the programbox or progressbox widget; each of the external programs will be started as a child process of C1, and C1 should wait(2) for the completion of every one of them; - in the main process, run d.programbox() or d.progressbox(), where d is your Dialog instance. This will last as long as there is at least one writer that hasn't closed the writing end of the pipe (a file descriptor can be duplicated with dup(2) [os.dup() in Python] and friends, as well as inherited by child processes upon fork(2) [os.fork() in Python]; each of these operations increases the "reference count" of the fd, which determines when the process reading from the pipe will see EOF instead of blocking in the read(2) system call, because this is when the "reference count" of the writing end of the pipe drops to 0 and all data previously in the pipe has been read that the reader process is told "there is no more data, this is over" [EOF]). Then your main process can wait for the child process C1 to exit. [1] An integer, i.e., low-level (OS-level) "file", different from a Python file object. It is often a good idea to close the file descriptors you don't need in the child, e.g., the reading end of the pipe in C1. Sometimes, it is even necessary, otherwise some operation may block (writing end of the pipe in the main process, that would make the dialog reader process believe, if the fd were not closed, that there is still the possibility of data being sent through the pipe). Closing a file descriptor in a child does not close it for the parent; it just closes a duplicate fd, decrementing its "reference count". This is exactly what MyApp.progressboxoid() in demo.py does, except that instead of spawning child processes with their stdout and stderr file descriptors assigned to the writing end of the pipe, it creates a Python file object from the file descriptor corresponding to the writing end of the pipe and writes to it as with any file object. This is the following part of MyApp.progressboxoid(): # Python file objects are easier to use than file descriptors. # For a start, you don't have to check the number of bytes # actually written every time... # "buffering = 1" means wfile is going to be line-buffered with os.fdopen(write_fd, mode="w", buffering=1) as wfile: for line in text.split('\n'): wfile.write(line + '\n') time.sleep(0.02 if params["fast_mode"] else 1.2) (os.fdopen() can be replaced with open() in not-too-old Python versions, but beware, arguments may vary slightly) In order to start a child process from C1, running an external program with its stdout and stderr redirected to the writing end of the pipe, you could use something like: class YourException(Exception): # Name it appropriately (UnableToRunBlaBla...) pass program = "systemctl" args = [program, "status", "foobar.service"] try: with subprocess.Popen(args, stdin=subprocess.DEVNULL, stdout=write_fd, stderr=subprocess.STDOUT, universal_newlines=True) as proc: out = proc.stdout.read() except OSError as e: raise YourException( "unable to find or execute {0!r}".format(program)) from e where 'write_fd' is the file descriptor corresponding to the writing end of the pipe, cf. MyApp.progressboxoid(). Then you can inspect proc.returncode to determine the exit status of the systemctl child process (if proc.returncode >= 0), or the signal that killed him, if any (this is if proc.returncode < 0; the signal number is -proc.returncode in this case). There are very slightly shorter alternatives to subprocess.Popen() in the subprocess module that are convenient in simple cases (run() in Python 3.5 and later; otherwise, call(), check_output(), etc. but these seem to be deprecated in Python 3.5). Anyway, subprocess.Popen() is the most powerful one used as a basis for the others, it is not even difficult to use, and not deprecated either. I suggest you study MyApp.progressboxoid() and tell me precisely which lines you don't understand. This may seem a bit complex at first, but it is robust and uses resources efficiently. If you use pipes to transfer the output of external programs to dialog, this output won't accumulate in memory and your processes can run for years producing huge amounts of data without any problem. This is essentially using the basic mechanisms provided for this purpose by the OS (i.e., fork(2), exec(3) and pipe(2)/pipe(7)). It is likely that the external programs you'll want to run will differ very little if at all in the way they need to be handled. Thus, you'll probably be able to use a common function or method to handle all of them (this is essentially what pythondialog does in dialog.py for the myriad of possible dialog commands, otherwise it would be totally unmaintainable). Note: you mentioned 'top'; in its default mode of operation, its output is probably unsuitable for sending to a pipe (except if it behaves differently when its stdout is not a terminal, this is possible). Probably something like "batch mode" (top's -b option) would be appropriate. If you've followed this advice and still didn't manage to get something working, post a minimal example of what you have tried. > Would you be able to guide me on how best to do that and some sample code > if possible? I looked at the demo.py and couldn't quite figure out the code > to do it there. My python is still beginner-ish but very keen to learn. > > Any help appreciated HTH :-) -- Florent |