Read Me

OVERVIEW
================================================================================

The Universal Multi-Threaded Server allows developers to focus on the
core functionality required to implement a project in a high performance
environment.

It has been said in writing software 80% of the time will be spent on
20% of the code. This is probably reasonably accurate. It follows then
that 20% of the time will be spent on 80% of the code. That is probably
also reasonably accurate.

The majority of software development involves basic control and message
code designed primarily to deliver data to the functional subroutine(s)
and pass the results on to other subroutines (or save them in a persistant
store ) for additional processing.

The goal of the UTMS is to provide that 80% of code and thereby reduce
the development cycle by 20%. A secondary design goal is to provide
a text based mechanism for deploying functional modules in a dynamic
processing chain using existing compiled dynamic libraries.

The developers writes application code, starting with prototype test code
running against stubs built in prototype production libraries according
to sound Agile practices.

These stubs are then chained together into a processing stream by editing
entries in the UMTS XML file. The new stream or daemon process is now ready
to be integrated into startup as well as the QA and test process.

It even masquerades as the target process in the process tables displayed by
the 'ps and 'top' and similar commands.

A default relay stub that transfer data is provided to allow rapid
initial deployment until development thread code lines and stubs and
ready to be integrated. Processing simulation ( for early testing ) or
debugging code modules can be deployed as soon as they are available.
Within a matter of hours a new runtime process linking development code
out of production dynamic libaries can be ready and running.

New functions may be added, existing functions replaced or relocated simply
by choosing different configuration files at start up. 

The UMTS is appropriate for rapid prototyping of problems suitable for
processing in an asynchrounous message passing multi thread architecture.
The UMTS provides a complete implementation of thread, semaphore,
message queue and mutex management.

This requires the libxml parser library.

UNIVERSAL MULTI-THREADED SERVER
================================================================================

The Universal Multi-Threaded Server provides a production grade execution
environment for code threads. These threads are developed independently
and located in dynamically linked libraries available at runtime.

With the UMTS a developer can chain together an arbitrary number of
pre-written code modules to construct a new processing chain simply by
editing an XML format configuration file and starting the binary. No
need for further compiling or linking.

An XML file contains information about each thread such as how many
processing threads of each type to create and how the thread processing
pipeline is constructed to transfer data between the threads.  This file
is read by the UMTS at startup and the corresponding thread processing
chain is created and executed.

The thread functionality is exposed via an simple api called from within
the UMTS processing infrastructure.

There are 8 points at which a code module and dynamic library may be
integrated in the XML file.

char *init_function; /* function which handles the thread init */
int (*init_ptr)(int, char **); /* called once with argc,argv */
char *cleanup_function; /* function which handles the thread close */
int (*cleanup_ptr)(int); /* called once with handle returned by init */
char *run_function; /* function which handles the thread runtime */
int (*run_ptr)( int, const u_char *,int *,u_char **, void *);
char *free_function; /* function which frees the opaque data */
void (*free_ptr)(void *); /* thread specific opaque data free */
char *open_function;
int (*open_ptr)(const char *stream, int flags); /* open opaque data stream */
char *close_function;
int (*close_ptr)(int handle); /* close opaque data stream */
/* function which reads the output stream */
char *read_function;
/* thread specific data read */
void (*read_ptr)(int handle,const void *buf,size_t count);
/* function which writes the output stream */
char *write_function;
/* thread specific opaque data write */
void (*write_ptr)(int handle,const void *buf,size_t count);

A code thread may be developed in any language which creates object modules
and dynamic libraries compatible with a C language environment.

Each functional thread has an input queue and an output queue under the
control of a queue service manager or QSM. These users thread dequeue
a record and call the 'run' function defined for this thread.

In the operational runtime thread a call is made to the run function when a
new data record has been received.

The code is called once for each data record with 5 arguments:

The data length - length of the input data as an unsigned integer
The data - a const u_char * to the input data
The opaque data length - a pointer to the length of the user data
The opaque data - a derefenced pointer to the user data if any
The queue service manager mask - a pointer to the QSM mask artifact

The runtime thread can assign or modify the opaque data and set the QSM
mask artifact. The opaque data is free by the 'free function' defined
in the XML, the allows application threads to use nested pointers and
other memory dereferences with confidence that the memory will not 'leak'
when the record processing is complete and the record data is released.


The run function returns one of the defined QSM state constants that will
control disposition of the data record:

QUEUE_STATE_OK /* send to next thread in processing chain */
QUEUE_STATE_ERROR /* invalid data - send to error thread for processing */
QUEUE_STATE_FAILED /* validation failed - send to error thread for processing */
QUEUE_STATE_DONE /* record processing complete - send to writer thread */
QUEUE_STATE_REJECTED /* doesn't meet criteria - send to reject thread */

Based on the record qsm state the queue service manager will dispatch the
record to the appropriate downstream processing thread. The default bindings above may be overridden by on queue state bindings in the XML file.

The dynamic binding allows records to be filtered dynamically to condition
An individual processing thread has no knowledge of the processing pipeline
and isn't involved in data or queue management. It does whatever it is
supposed to do and releases the record when it is done knowing that the result
will be processed.

In more advanced applications the queue disposition is controlled by the QSM
state and the QSM state mask which allows the application code to commuicate
additional state transitition information in the form of a bit mask.

Individual threads may maintain an ordered or unordered queue and the
processing stream may be throttled by management of the number of
processing threads.

Input Stream Processing
===============================================================================

The Universal Multi-Threaded Server reads input data which is broken
into user defined records. Records may defined in terms of a fixed record
size ( with an optional data offset and data size ), a starting and ending
record signature or a text based file suitable for input with fgets.

Input to the UMTS comes from the standard input or an optional file based
interface. The developer may specify a default input routine and output
routine to support non-file based i/o streams.

The input stream does not have to be a fixed multiple of any sizes and
any partial records at the end may be preserved or discarded as specified
in the XML file. Once all data has been read from the input device an End
of File packet is routed through the threads.

=========


There is also a verbose option which will report progress information
on stderr

A partial list of options and defaults is available using: umts -h

It should build and run on a POSIX compliant computer with the GNU
toolchain. A binary built on Fedora 7 with gcc version 4.1.2 20070925
(Red Hat 4.1.2-27) is included:

Quick Start:

to build:
	cd src; make # build it
	make test # run the validation tests

to run:
	cd src; make test # verify that the binary functions properly


The test target runs two simple tests. The first test uses a test data
file and the relay stubs to verify dynamic binding and data relay through
the processing stream

which is
composed entirely of null characters. This key is contained in the file
'null.key'.

 This ensures that the input text is not transformed by the encryption
 and provides a simple mechanism to verify that the text is being
 passed through intact. The results of the test are written to a file
 and compared against the input text with cmp.

If this test succeeds an encryption test is performed. This test encrypts
the input data using the simple test key 'FEEDBEEF' ( found in the
file test.key ). The results are compared the expected values store in
test.test.key.expected with cmp.