REX : Remote EXecution Services : rfork Code
Status: Pre-Alpha
Brought to you by:
jvdias
File | Date | Author | Commit |
---|---|---|---|
dox | 2009-05-10 | jvdias | [r3] 2009-05-01 JVD : |
src | 2009-05-10 | jvdias | [r4] JVD 2009-05-01 Oops ! do define the SINGLE work... |
ChangeLog.2009-04-01 | 2009-04-12 | jvdias | [r1] |
README | 2009-04-19 | jvdias | [r2] 2009-04-19 JVD |
RELEASE_NOTES | 2009-04-12 | jvdias | [r1] |
~~~~ REX: Remote Execution Services ~~~~ ~~~~ Build & Installation Guide ~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1. Having obtained and unpacked the tarball containing this document, eg. from: https://librex.sourceforge.net/librex-${REX_VERSION}-${REX_RELEASE}.tar.bz2 , ( REX's VERSION is currently 1.0.0 and its RELEASE is "beta-1" (@2009-04) ) as a user or super-user of a Linux or Solaris host, run these commands: $ $UNZIP < $REX_TARBALL | tar -xpf - $ cd REX-${VERSION} # ^- The current directory here ($CWD/getcwd()) is known as the # "REX Build Directory" . $ ./configure $ make && make install # ^- install accepts the ${DESTDIR} setting to select # the root directory of the installation . # other make targets: rpm , tar , dist, librex , include, edit, # clean, distclean OR : $ ./build && ./install OR : $ ./install OR $ ./release &&( bunzip2 < librex-${REX_VERSION}-${REX_RELEASE}.${REX_ARCH}.tar.bz2 | (cd $DESTDIR; tar -xpf -) ) where $UNZIP is 'bunzip2' if you received a bzcat(1) archive or 'gunzip' if you received a gzcat(1) archive - the parenthesized numbers after a name here refer to UNIX manual pages (for UNIX, read : Linux OR Solaris, unless otherwise stated), and such conventions are used throughout this document. N.B: REX does not necessarily use autoconf(1) or automake(1) facilities at all ; at the moment, the build is done with entirely with GNU make(1). Solaris users must have GNU make(1) 3.80+ installed as : /usr/sfw/bin/gmake (from Solaris' companion GNU tools CD) or /usr/local/bin/make (as installed from GNU Make 3.81 from SunFreeWare.com). Users can easily modify the make(1) variable settings in the REX Build directory configuration make file: ./config.mk to customize REX build and installation; but this should be infrequently required. Package Build Requirements: GNU gcc - tested with : 3.46, 4.1.2, 4.2.4, 4.3.2, 4.3.3, 4.4.0 optional: GNU g++ / c++ 4+ if libREX C++ library is being built GNU binutils OR Solaris /usr/ccs/bin/{ld,as} linker/loader/assembler tools GNU make(1) - 3.80 or higher - 3.81 recommended GNU bash(1) OR Solaris bash(1) OR AT&T ksh(1) shell GNU coreutils or Solaris coreutils PERL 5.8.2+ Optional Package Features and Requirements : o Configuration & Log Relational SQL Database : Requires either PostGreSQL, MySQL, Oracle, Informix, DB/2, Daytona o XML Configuration and Log database: Requires libxml2 OR libexpat o DocBook-XSL-Stylesheets IFF documentation is re-generated with $ make dox o RPM on Linux - rpmbuild utilities IFF $ make rpm is used. After installation, as the REX user who wishes to enable a new REX PROGRAM , ensure you have at least once run the $ rex_key --create command, and then the $ rex_key --distribute command, to distribute the REX keys for this host and user to each host in the active REX configuration file "servers" option ; these hosts form a "cluster" or "grid" of participating hosts, the least-loaded or user selected of which can process and/or originate REX TRANSACTIONS such as rfork(), rexecve(), or rcall() . You can also use $ rex_key -c(reate) -p(rogram) $MY_PROGRAM To create an optional "program key" that can can also be an authorization target; ie. the entity identified by ${HOST}.${USER}[.${PROGRAM}] # ${PROGRAM} being optional must identify an AUTHORIZED USE of the REX system; access to each host, user, and program CAN be granted or revoked by the REX user that owns the ORIGIN instance of the program, or the super-user IFF REX was installed as root. REX does NOT have to be installed as root; if not installed as root, that installation is an ORIGINATOR USER installation ; if installed as root, then each user on whose behalf the root instance of rexd fork(2)s to create has one REX SESSION CONTROLLER process for that user. Use of Session Controllers enables file descriptor passing from a single server that accept(2)s incoming TCP connections to clients that are identified by a ${HOST}.${USER}.${PROGRAM} triple. So there are 8 possible types of REX process : o Parent or "Controller" Processes: 1. The single root Daemon rexd instance ( optional: only if REX installed as root ), that implements "REX TRANSACTIONS" : - rfork() : fork(2) replacement : create new process similar to fork(2) on remote host - rcall() : remote function call: send input stack frame of size InSz bytes, enter remote code, and receive output stack frame of size OutSz bytes - rjump() : remote entry : send input stack frame (optional) , enter remote code, possibly with rflags set to "Asynchronous: do not wait", and possibly NOT returning. - "I/O Transactions" : these are handled by instances of the Stub / Proxy Process, which provide Ancillary functions for Signal Handling, IPC, and Remote Resource Access (files, directories, fifos, pipes, sockets, semaphores, message queues) . Thus, for instance, an rfork()-ed process can continue to read the same FDs and Sockets it had open at time of rfork() on the originator host when running on the executor host; only each such FD will now be a socket. 2. Session Controller :A non-root rexd "session controller" instance 3. Controller : A non-root non-session controller instance (eg. stub process) o 4. "Local Stub" or "Standin" / Proxy and I/O Controller process ( never a parent! ) o Children or "Controlled" Processes of one of the above parents 5. A "Restored" or "rfork()" child process 6. A "Call Controller" process 7. The "Remote Stub" I/O Controller / Proxy process 8. A "Remote Execve" process or remote execution of program. Here is an example "tutorial" "baseline" test program using rfork() : Example #1: /* * test_rfork_1.c : simple fork() of process to run on least-loaded host */ #include <stdio.h> #include <rex.h> int main( int argc, char **argv, char **envp ) { printf("parent %u runs on host %s - load: %g\n", (unsigned)getpid(), gethostname(), rload() ); pid_t pid = rfork(); if( pid == 0 ) printf("child %u parent %u origin pid %u origin parent %d origin host %s runs on host %s - REX loadavg: %g\n", (unsigned)getpid(), (unsigned)getppid(), (unsigned)rgetpid(), (unsigned)rgetppid(), rgethostname, gethostname(), rload() ); else if ( pid > 0) wait(pid); else return(errno); return(0); } $ make -f ${REX_INSTALL_DIR}/Makefile test_rfork_1.o When the "./rex.rc" file contains : "servers : my_linux_x86_64_pc_1, my_linux_x86_64_pc_2 clients : localnets " When the command is run on host my_linux_x86_64_pc_1: $ ./test_rfork_1 This would print out: "parent PPPPP runs on host my_linux_x86_64_pc_1 - load 0.fffffffff+-eee child CCCCC parent ppppp origin pid cccccc parent PPPPP origin host my_linux_x86_64_pc_1 runs on host my_linux_x86_64_pc_2 - REX loadavg:0.FFFFFFFFF[+-]EEE " where : PPPPP is the pid of the Originator Parent on the Origin Host (my_linux_x86_64_pc_1) CCCCC is the pid of the Restored Process on the Executor Host (my_linux_x86_64_pc_2) ccccc is the pid of the Stub process on the Origin Host ppppp is the pid of the Controller process on the Executor host 0.fffffffff+eee is the fraction and exponent of a (double) floating-point number representing REX's perception of the Origin Host's average load 0.FFFFFFFFF+EEE is the fraction and exponent of a (double) floating-point number representing REX's perception of the Executor Host's average load The remote child's standard input and output terminal device file descriptors are re-directed to TCP sockets connected to the local Stub / Proxy process running on the origin host which writes / reads them respectively to the real terminal inherited from the originating parent, and is wait(2)-ed for by the origin parent. No ptrace() control is retained over the remote restored child because the process' have no shared inherited IPC or regular file or semaphore or message queue or socket resources. NOTE: If you have not previously transferred a process to a remote host that is going to execute its code, the code files will be transferred into a per-host subdirectory of the "rex-temp-directory" ( /tmp/.rex_${host}_${user} by default), so the initial run can take much longer than subsequent runs; automatic file transfer can be independantly enabled / disabled or set to use SSH or rsync or some other program with the "rex-auto-transfer" configuration directive. NOTE: the performance of subsequent runs of this process on your system indicates the baseline "REX Overhead" load that every REX using process will incur; this is typically very light; for example, on 2 2GHz CPU hosts on a 1Gbps LAN , the time(1) command reports an elapsed time of @ 1.0-3 seconds of which user CPU @ 20% system @ 80% and a total data transfer of less then 30Kb ( REX transfers only those parts of a running process that differ from copies on the remote host; so if the remote host has access to a copy of the test_rfork_1 executable, and all shared libraries mapped into the origin process, this transfer can be greatly optimized . But this is greater than the time taken for a local fork(2) operation; thus ONLY if your child process is going to do alot more work than this baseline process, will any load-balancing advantage be gained by using rfork(). The advantage that can be gained by using rfork() instead of fork() for highly loaded worker child processes can be substantial for expecially hard-working children, since if only the least-loaded hosts amongst a cluster of hosts take on large work, then then average performance of all hosts in the cluster will increase; the originator host does not experience the load itself, and its performance is thereby increased. The local stub/proxy process left on the origin host as a standin that can be wait(2)-ed for and kill(2)-ed, implements a blocking select(2) loop and is consistently rated amongst the least user CPU or system CPU or memory intensive processes by top(1) and rtop(1) . A REX Call Controller Process embodies an instance of a program or shared library that allows itself to be called / entered with an identifying ${HOST}.${USER}.${PROGRAM}.${FUNCTION} quad from a rexcd instance; on program startup, a call controller creates a queryable database of Remote Entry Points, as directed by configuration directives / API calls, that map input and output REX STACK FRAME definitions to ${FUNCTION} names provided by the program; remote callers can then send input stack frames and receive output stack frames to implement: "Remote Entry" : (rjump : send input, enter and possibly never return) or "Remote Function Call" (rcall : send input parameters, enter, and receive output) operations . REX Remote Entry Points (REPs) which can be the targets of the rcall() or rjump() operations can be defined by any of : o The output of the "repdb" program when invoked with an input parameter (or file on stdin) of the ELF shared object containing the entry points and one of the following argments -g : use '-g' information in shared object to emit stack frame info -x : use xml entry point definitions - see 'rex.dtd' . -d : use database definition : "${HOST}.${USER}.${DBMS}.${DB_NAME}" which must have a "REXP" table that maps function name to correct input and output stack frame sizes : a tuple: struct rex_entry_point_s { const char *name, *elf_file, *host, *user, *program, *config; void *address; struct rex_stack_s { unsigned size; } input, output; } ; NOTE : REX IS NOT RPC !! REX makes no use of Sun RPC - REX implements its own highly simplified version of RPC which relies on the user to send a correct stack frame, and does NO XDR - parameter marshalling / LSB <-> MSB translation of parameters - so if you are attempting to send MSB parameters to a LSB host or vice versa, you must use htonl / htons et al to perform the parameter conversion. Later versions of REX may do XDR, but REX is primarily designed for use between HOMOGENOUS hosts only at the moment so the question does not arise . NOTE: It is strongly advised for users NOT to invoke "repdb" WITHOUT the "-g" option; ordinarily, users should use: $ repdb -gd < $my_executable | $MY_SQL_PARSER OR $ repdb -gx < $my_executable > ${my_executable}.xml Users can later run binutils' strip(1) program on the executables and REX will successfully use the stack frame definitions in the SQL or XML databases to invoke functions in such strip(1)-ed shared objects . But users CAN write their own XML entry point definitions or insert entries in the SQL entry point database by invoking "repdb" WITHOUT "-g" ; users should view this as akin to poking values into an executable file - not advisable, but possible! "The Active REX Configuration File" can be either: 1. ./rex.rc 2. ~/rex.rc 3. ${REX_ROOT}/etc/rex.rc where REX_ROOT is by default '/'. Configuration files can direct the REX library to read the rest of the configuration from an XML file or database if XML or database configuration is enabled; all REX programs & the internal API can be directed to take configuration from flat ".rc" files OR database OR XML files. Configuration files can also be pre-processed by cpp(1) and can use '#include' statements. See the REX manual pages for more details: o REX.7 - Introduction o REX_install.8 - Build & Installation Guide - this document o REX_config.8 - Configuration Guide o rexd.8 - REX Daemon Reference o REX.3 - REX Programming Guide o r{,ex_}*.3 - REX API Reference o REX.1 - REX Tools & Utilities Guide o r{,ex_}*.1 - REX Tools & Utilities Individual Manual Pages REX manual pages can be re-generated with $ make dox/{man,html,pdf}/man[1-8]/${PAGE} in the REX build directory.