From: Marco B. <em...@be...> - 2002-08-31 19:34:42
|
brief introduction to the new function POSIX:EXEC (bad name but that's ok for now) - spawns sub process (fork/execve). takes 5 arguments: - name of file to run, a string. - list of arguments to pass, a list of strings. - whether to wait for sub process to terminate or not - file descritors to stream mapping. a list specifying which file descrptor (a number) to attach to which stream. the stream can be an already opened stream, or a list containg :DIRECTION, :BUFFERED, :ELEMENT-TYPE and :EXTERNAL-FORMAT keyword args with same semantics as OPEN. - enviroment to pass to sub process, a hash table with variable names as keys and values as strings. (once this works i'll add ability to specify enviroment as an alist, a plist, or an array of char arrays). returns multiple values, the first is the sub process's return value (or NIL if we didn't wait for it to terminate). the remaing values are the streams specifed for the file desriptors. would it better to return a single list containing these things instead? it does not spawn a shell (unlike MAKE-PIPE-INPUT-STREAM and MAKE-PIPE-OUTPUT-STREAM) , this because i don't see any advantages to it and spawning /bin/bash (or /bin/tcsh) just so that the shell can then fork and execve seems like a waste (if i've missed something tell me). currently it allows a lot of control over the subproc at the expense of not being very user freindly, ie: the first argument is not automatically set to the file's name, $PATH is not searched in the case of non absolute file names. i plan on building a convenient program calling/chaing macro on top of this function. any thoughts/comments on the road i've taken so far? however, i've got a problem (surprised? :)): currently the only streams you can pass it (to attach to file descriptors) are streams attached to files because only thoses streams have an associated Handle. I want to be able to attach any stream, for example, attach a string-output stream to standard out. my current idea is to just read/write from/to the stream to/from the sub process, but i think this has some major dead lock problems. any ideas? i'm not offering a patch yet because a) i'd like to test some things a bit more and b) finish some things which aren't working (like this stream problem and setting up the sub proc's enviroment). -- -Marco Ring the bells that still can ring. Forget your perfect offering. There is a crack in everything. That's how the light gets in. -Leonard Cohen |
From: Sam S. <sd...@gn...> - 2002-09-03 15:55:11
|
> * In message <m28...@be...> > * On the subject of "running sub process" > * Sent on 31 Aug 2002 21:35:19 +0200 > * Honorable Marco Baringer <em...@be...> writes: > > brief introduction to the new function POSIX:EXEC (bad name but that's > ok for now) - spawns sub process (fork/execve). takes 5 arguments: I prefer that you make the existing functions (EXT:EXECUTE on UNIX & AmigaOS and EXT:SHELL on win32 and AmigaOS) deprecated and make them call your new function. > - name of file to run, a string. > - list of arguments to pass, a list of strings. > - whether to wait for sub process to terminate or not > - file descritors to stream mapping. a list specifying which file > descrptor (a number) to attach to which stream. the stream can be > an already opened stream, there should be 4 arguments: ((:standard-ouput *standard-output*) *standard-output*) ((:standard-input *standard-input*) *standard-input*) ((:error-ouput *error-output*) *error-output*) and this file-descriptor <--> stream map (which I am not sure is a good idea anyway, but, if you want it...) > or a list containg :DIRECTION, :BUFFERED, :ELEMENT-TYPE and > :EXTERNAL-FORMAT keyword args with same semantics as OPEN. no! the user can open the stream himself > - enviroment to pass to sub process, a hash table with variable > names as keys and values as strings. (once this works i'll add > ability to specify enviroment as an alist, a plist, or an array of > char arrays). > > returns multiple values, the first is the sub process's return value > (or NIL if we didn't wait for it to terminate). the remaing values are > the streams specifed for the file desriptors. would it better to > return a single list containing these things instead? a single list is never a good idea. > it does not spawn a shell (unlike MAKE-PIPE-INPUT-STREAM and > MAKE-PIPE-OUTPUT-STREAM) , this because i don't see any advantages to > it and spawning /bin/bash (or /bin/tcsh) just so that the shell can > then fork and execve seems like a waste (if i've missed something tell > me). good. then MAKE-PIPE-*-STREAM should call this new function of yours. > currently it allows a lot of control over the subproc at the expense > of not being very user freindly, ie: the first argument is not > automatically set to the file's name, $PATH is not searched in the > case of non absolute file names. i plan on building a convenient > program calling/chaing macro on top of this function. > > any thoughts/comments on the road i've taken so far? there should be a lisp variable, initialized from $PATH, that _is_ searched by this function. > however, i've got a problem (surprised? :)): currently the only > streams you can pass it (to attach to file descriptors) are streams > attached to files because only thoses streams have an associated > Handle. I want to be able to attach any stream, for example, attach a > string-output stream to standard out. my current idea is to just > read/write from/to the stream to/from the sub process, but i think > this has some major dead lock problems. any ideas? I suggest that you ask this on comp.unix.questions. -- Sam Steingold (http://www.podval.org/~sds) running RedHat7.3 GNU/Linux <http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/> <http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html> Warning! Dates in calendar are closer than they appear! |
From: Marco B. <em...@be...> - 2002-09-04 21:19:34
|
Sam Steingold <sd...@gn...> writes: > this file-descriptor <--> stream map (which I am not sure is a good > idea anyway, but, if you want it...) as Pascal Bourguignon pointed out, there are (admittedly rare) cases when this generality is usefull; that and the fact that i've already written the code to deal with it... > > or a list containg :DIRECTION, :BUFFERED, :ELEMENT-TYPE and > > :EXTERNAL-FORMAT keyword args with same semantics as OPEN. > > no! the user can open the stream himself good, i was hoping you'd say that (because then i'd be done with this patch). but if i want a stream attached to the output of, say, "gpg --unencrpyt", how do i open the stream i then pass to the function? make-pipe-output-stream, the first i looked at when writing this patch, is written in .d and uses pipe(2). ---- Here's where i'm going with this function: POSIX:EXEC (or EXT:EXECUTE or whatever) is an interface to the operating system's low-level process spawning mechanisms (execve(2) and whatever windows/amiga uses). it covers all the weird cases, like when argv[0] is not the program's name, or when i don't want to search in the path or when i want file descriptor voodoo majik. If you don't want to worry about these details or the "common case" is good enough we'll still provide nice functions for easily running sub process. Therefore i would not deprecate EXT:RUN-PROGRAM or EXT:RUN-SHELL-COMMAND as they're usefull abrevviations, and i'll add something like scsh's run function when this patch is ready (and a nice lispy $PATH searching mechanism). -- -Marco Ring the bells that still can ring. Forget your perfect offering. There is a crack in everything. That's how the light gets in. -Leonard Cohen p.s. - i have designed this function with execve(2) in mind, if windows or amiga have slightly different functions (example: maybe on windows you can specify the program's initial pwd) i'd appreciate it if someone told me so i could try and fake this in the UNIX version. |
From: Sam S. <sd...@gn...> - 2002-09-04 21:34:29
|
> * In message <m2a...@be...> > * On the subject of "Re: running sub process" > * Sent on 04 Sep 2002 23:20:18 +0200 > * Honorable Marco Baringer <em...@be...> writes: > > but if i want a stream attached to the output of, say, "gpg > --unencrpyt", how do i open the stream i then pass to the function? > make-pipe-output-stream, the first i looked at when writing this > patch, is written in .d and uses pipe(2). so what is the question? how to pass, say, a STRING-OUTPUT-STREAM as the *STANDARD-OUTPUT*? I suggested earlier that you ask this on comp.unix.questions. Did you? The only suggestion I have is to use pipe() and threads. > i would not deprecate EXT:RUN-PROGRAM or EXT:RUN-SHELL-COMMAND as > they're usefull abrevviations, and i'll add something like scsh's run I never said that these should be deprecated. I did say that the old primitives in pathname.d should be superseded by your new function. > function when this patch is ready (and a nice lispy $PATH searching > mechanism). see my previous message on how $PATH should be handled. -- Sam Steingold (http://www.podval.org/~sds) running RedHat7.3 GNU/Linux <http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/> <http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html> Do not tell me what to do and I will not tell you where to go. |
From: Marco B. <em...@be...> - 2002-09-04 21:50:26
|
Sam Steingold <sd...@gn...> writes: > > but if i want a stream attached to the output of, say, "gpg > > --unencrpyt", how do i open the stream i then pass to the function? > > make-pipe-output-stream, the first i looked at when writing this > > patch, is written in .d and uses pipe(2). > > so what is the question? what type of stream do i pass to the function if i want to read from a sub process' standard output? (ignoring the string-stream question for now) in this snippet: (setf foo (make-???-stream :direction :output)) (posix:exec command" :file-descriptors '((1 foo))) (setf command-output (read-line foo)) what type of stream should foo be? -- -Marco Ring the bells that still can ring. Forget your perfect offering. There is a crack in everything. That's how the light gets in. -Leonard Cohen |
From: Sam S. <sd...@gn...> - 2002-09-04 22:11:53
|
> * In message <m24...@be...> > * On the subject of "Re: running sub process" > * Sent on 04 Sep 2002 23:51:07 +0200 > * Honorable Marco Baringer <em...@be...> writes: > > Sam Steingold <sd...@gn...> writes: > > > > but if i want a stream attached to the output of, say, "gpg > > > --unencrpyt", how do i open the stream i then pass to the function? > > > make-pipe-output-stream, the first i looked at when writing this > > > patch, is written in .d and uses pipe(2). > > > > so what is the question? > > what type of stream do i pass to the function if i want to read from a > sub process' standard output? (ignoring the string-stream question for > now) > > in this snippet: > > (setf foo (make-???-stream :direction :output)) > (posix:exec command" :file-descriptors '((1 foo))) > (setf command-output (read-line foo)) > > what type of stream should foo be? this is not what we want. (ext:execute "file" :output :stream) will return a stream which will contain the standard output of the "file". generally, (ext:execute "file" :arguments (arglist nil) :search-path (path *system-search-path*) :standard-output stream-spec :standard-input stream-spec :error-output stream-spec :wait wait-p :more-file-descriptors fd-map) ==> status input-stream-1 output-stream input-stream-2 * arguments: - file: program file to run - arguments: a list of strings (argv) - search-path: directory list to search for "file", should default to the value of a lisp variable which is initialized at start time based on $PATH - standard-output, standard-input, error-output: the stdio of the new process; can be: NIL (discarded), pathname designator (use this file), stream (use this stream), number (use this file handle). The first version of the patch may limit STREAMs to handle-streams (i.e., those associated with a handle) - wait: whether to wait for termination of not - more-file-descriptors: the map FD<-->stream you want. may be (should be!) omitted in the first patch * return values: - status: error code (when wait-p is T) or PID (when wait-p is NIL) - input-stream-1: the standard output for the new process - output-stream: the standard input for the new process - input-stream-2: the error output for the new process Example: (eq foo (nth-value 1 (ext:execute "..." :standard-output foo ...))) should be T. -- Sam Steingold (http://www.podval.org/~sds) running RedHat7.3 GNU/Linux <http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/> <http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html> I don't have an attitude problem. You have a perception problem. |
From: Arseny S. <am...@ic...> - 2002-09-04 22:32:40
|
Hello Marco, Thursday, September 05, 2002, 7:20:18 AM, you wrote: > Here's where i'm going with this function: POSIX:EXEC (or EXT:EXECUTE > or whatever) is an interface to the operating system's low-level > process spawning mechanisms (execve(2) and whatever windows/amiga > uses). Windows haves execve too. int _execve( const char *cmdname, const char *const *argv, const char *const *envp ); I'm not sure about in/out/error redirection for execve, I think it's impossible in windows. I suspect that current implementation uses CreateProcess instead of exec[v[e]] because of stream redirection. CreateProcess takes i/o/e handle parameters as well as a lot of other parameters. It will be too cruel to post CP description here, just need to remember that: CP doesn't search PATH CP takes enviroinment in form "name=val\0name1=val1\0\0", but can leave same enviroinment. CP takes current directory parameter for new process. CP takes 3 file handles (HANDLE). I remember, that clisp streams for win32 are based on HANDLEs too (not on int descriptors). -- Best regards, Arseny |
From: Sam S. <sd...@gn...> - 2002-09-05 13:46:52
|
> * In message <934...@ic...> > * On the subject of "Re[2]: running sub process" > * Sent on Thu, 5 Sep 2002 09:35:36 +1000 > * Honorable Arseny Slobodjuck <am...@ic...> writes: > > CP takes 3 file handles (HANDLE). I remember, that clisp streams for > win32 are based on HANDLEs too (not on int descriptors). IIUC, HANDLE is just another name for an int, just like FDs on Unix are not really ints but "fd_t" (at least on some systems). -- Sam Steingold (http://www.podval.org/~sds) running RedHat7.3 GNU/Linux <http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/> <http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html> Lisp is a way of life. C is a way of death. |
From: Marco B. <em...@be...> - 2002-09-05 00:14:40
|
Sam Steingold <sd...@gn...> writes: > > what type of stream do i pass to the function if i want to read from a > > sub process' standard output? (ignoring the string-stream question for > > now) > > > > in this snippet: > > > > (setf foo (make-???-stream :direction :output)) > > (posix:exec command" :file-descriptors '((1 foo))) > > (setf command-output (read-line foo)) > > > > what type of stream should foo be? > > this is not what we want. > > (ext:execute "file" :output :stream) > > will return a stream which will contain the standard output of the > "file". ok, i misunderstood you. however, these questions remain: should the streams be buffered? what :element-type and :external-format? so i'd need something like: (ext:execute "foobar" :output (:stream :buffered nil :external-format :mac)) while we're here: EXT:RUN-PROGRAM accepts :TERMINAL for :INPUT and :OUTPUT, is this still neccessary since we can now pass *STANDARD-OUTPUT*/*STANDARD-INPUT* directly? > - more-file-descriptors: the map FD<-->stream you want. > may be (should be!) omitted in the first patch i could omit the parameter, but the :standard-input, :standard-output and :standard-error parameters are implemented in terms of the file-descriptor mapping code, so don't understand why. > * return values: > > - status: error code (when wait-p is T) or PID (when wait-p is NIL) i find this very confusing. why not return the sub proc's status as the first value, NIL if wait-p is NIL and PID as the second return value? since CLISP has no equivalent to wait(2) or kill(2) i don't really know what use there is to getting the sub proc's PID, but it seems like a nice idea anyway. > - input-stream-1: the standard output for the new process > > - output-stream: the standard input for the new process > > - input-stream-2: the error output for the new process > > Example: > > (eq foo (nth-value 1 (ext:execute "..." :standard-output foo ...))) > > should be T. wouldn't it be better to follow the UNIX convention of first input, then output, then error? which would make the example: (eq foo (nth-value 2 (ext:execute "..." :standard-output foo ...))) --- a few words on :search-path, :standard-output, :standard-input, :error-output, :more-file-descriptors and (the immaginary) :pwd. i think ext:execute is doing more than it should. it'd be nice to specify the pwd of the new process, this is an extra parameter to CreateProcess windows and an extra system call on unix, so should we add that into ext:execute? probably not. i have had the desire in the past to setup argv[0] as something other than the exec'd file's name, is this common? certainly not, but it should be doable if someone wants. what i'd really prefer to do is to have a function (ext:execve) with only the :file-descriptors, :wait, :environment and, of course, program arguments. there would be no path searching, no directory changing, and no messing of the arguments, these things would be added in EXT:EXECUTE (which at this point could happily lose the :more-file-descriptors argument and gain a :working-directory argument) and/or EXT:RUN-PROGRAM. This seperation would give us both flexibilty when we are willing to pay for it and convenience when we don't need flexibility, kind of like the current seperation between MAKE-PIPE-*-STREAM and RUN-PROGRAM. -- -Marco Ring the bells that still can ring. Forget your perfect offering. There is a crack in everything. That's how the light gets in. -Leonard Cohen |
From: Sam S. <sd...@gn...> - 2002-09-05 14:01:59
|
> * In message <m2e...@be...> > * On the subject of "Re: running sub process" > * Sent on 05 Sep 2002 02:15:31 +0200 > * Honorable Marco Baringer <em...@be...> writes: > > Sam Steingold <sd...@gn...> writes: > > > (ext:execute "file" :output :stream) > > > > will return a stream which will contain the standard output of the > > "file". > > ok, i misunderstood you. however, these questions remain: should the > streams be buffered? what :element-type and :external-format? so i'd > need something like: > > (ext:execute "foobar" :output (:stream :buffered nil :external-format :mac)) okay, you are right. > while we're here: EXT:RUN-PROGRAM accepts :TERMINAL for :INPUT and > :OUTPUT, is this still neccessary since we can now pass > *STANDARD-OUTPUT*/*STANDARD-INPUT* directly? not necessary, but a nice feature. > > - more-file-descriptors: the map FD<-->stream you want. > > may be (should be!) omitted in the first patch > > i could omit the parameter, but the :standard-input, :standard-output > and :standard-error parameters are implemented in terms of the > file-descriptor mapping code, so don't understand why. okay, keep it. > > * return values: > > > > - status: error code (when wait-p is T) or PID (when wait-p is NIL) > > i find this very confusing. why not return the sub proc's status as > the first value, NIL if wait-p is NIL and PID as the second return > value? since CLISP has no equivalent to wait(2) or kill(2) i don't > really know what use there is to getting the sub proc's PID, but it > seems like a nice idea anyway. wait(2) and kill(2) should be added too. Arseny, is there an analogue on win32? > wouldn't it be better to follow the UNIX convention of first input, > then output, then error? absolutely! > a few words on :search-path, :standard-output, :standard-input, > :error-output, :more-file-descriptors and (the immaginary) :pwd. you mean :cwd - change working directory :-) > I think ext:execute is doing more than it should. it'd be nice to > specify the pwd of the new process, this is an extra parameter to > CreateProcess windows and an extra system call on unix, so should we > add that into ext:execute? probably not. i have had the desire in the > past to setup argv[0] as something other than the exec'd file's name, > is this common? certainly not, but it should be doable if someone > wants. what i'd really prefer to do is to have a function (ext:execve) > with only the :file-descriptors, :wait, :environment and, of course, > program arguments. there would be no path searching, no directory > changing, and no messing of the arguments, these things would be added > in EXT:EXECUTE (which at this point could happily lose the > :more-file-descriptors argument and gain a :working-directory > argument) and/or EXT:RUN-PROGRAM. This seperation would give us both > flexibilty when we are willing to pay for it and convenience when we > don't need flexibility, kind of like the current seperation between > MAKE-PIPE-*-STREAM and RUN-PROGRAM. I prefer a single built-in which does everything related to process creation and all the other convenience functions implemented on top of that. That does not imply that the built-in will be a huge function. Subroutines can be abstracted &c... The rationale is that the interaction between different system calls and their behavior is not very portable and should be handled in one place. -- Sam Steingold (http://www.podval.org/~sds) running RedHat7.3 GNU/Linux <http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/> <http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html> MS DOS: Keyboard not found. Press F1 to continue. |
From: Arseny S. <am...@ic...> - 2002-09-06 00:07:58
|
Hello Sam, Friday, September 06, 2002, 12:01:55 AM, you wrote: >> i find this very confusing. why not return the sub proc's status as >> the first value, NIL if wait-p is NIL and PID as the second return >> value? since CLISP has no equivalent to wait(2) or kill(2) i don't >> really know what use there is to getting the sub proc's PID, but it >> seems like a nice idea anyway. > wait(2) and kill(2) should be added too. > Arseny, is there an analogue on win32? IIUC, WaitForSingleObject(hprocess, time) and TerminateProcess(hprocess, exitcode) match. -- Best regards, Arseny |