[XPS-devel] XPS In A Nutshell

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 422-6466

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
	<META HTTP-EQUIV=3D"CONTENT-TYPE" CONTENT=3D"text/html; charset=3Diso-8859=
-1">
	<TITLE></TITLE>
	<META NAME=3D"GENERATOR" CONTENT=3D"OpenOffice.org 1.0  (Linux)">
	<META NAME=3D"CREATED" CONTENT=3D"20021230;22521600">
	<META NAME=3D"CHANGED" CONTENT=3D"20021231;3503800">
	<STYLE>
	<!--
		H1 { color: #000080; text-align: left }
		H1.western { font-family: "Arial", sans-serif; font-size: 20pt }
		P.western { font-family: "Arial", sans-serif }
		H2 { color: #000080; text-align: left }
		H2.western { font-family: "Arial", sans-serif; font-size: 16pt }
	-->
	</STYLE>
</HEAD>
<BODY LANG=3D"en-US">
<P CLASS=3D"western" ALIGN=3DCENTER STYLE=3D"margin-top: 0.42cm; page-break=
-after: avoid">
<FONT COLOR=3D"#000080"><FONT FACE=3D"Albany, sans-serif"><FONT SIZE=3D6><B=
>XPS
In A Nutshell</B></FONT></FONT></FONT></P>
<P CLASS=3D"western" ALIGN=3DCENTER><I>Reid Spencer, January 1, 2003</I></P=
>
<P CLASS=3D"western">In reflecting on the past year's development of
the eXtensible Programming System (XPS) , I decided that a shorter
but comprehensive introduction to the system would be a good idea for
those unfamiliar with XPS. This paper provides a quick overview of
the system to help you get the &ldquo;big&rdquo; picture and the
vision for the project. Much of the content is written in the present
tense, as if XPS were a completed system. However, there is much to
do! If you think the goals of the system are worthy, I hope you'll
consider contributing your particular expertise to the development of
XPS.</P>
<H1 CLASS=3D"western">Background</H1>
<P CLASS=3D"western">The original idea for XPS came about many years
ago after watching the first few episodes of Star Trek: The Next
Generation (circa 1987). Like most software engineers at the time, I
was intrigued with the holodeck. But, my fascination with this
programmable virtual world quickly turned into an earnest exploration
of what it would take to build such a system. Instead of letting all
the worrisome physics get in the way, I looked at the holodeck as a
software system. What kind of software would be needed for such a
system? Certainly, it would have to be object-oriented as it dealt
mostly with the generation and manipulation of simulated real world
objects (now <I>there's</I> an oxymoron). Programming the holodeck
would have to be easy, since the Star Trek characters could create
amazingly complex worlds rather quickly. It would have extensive
command and control facilities as frequently exhibited on the show.
It would have to be aware of both the human and generated
participants in the scene, have some level of autonomy and provid
appropriate spontaneous character interactions and scene changes. It
would operate in at least four dimensions to maintain a continuous
virtual reality for the participants. It would both understand and
generate several natural language(s) fluently. It would incorporate
much of artificial intelligence. It would be fast, efficient, fault
tolerant, robust, dependable, secure, and safe. In essence, it would
combine everything everyone knows about software and then some!
(Sounds to me like a few large corporate IT projects I've worked on).</P>
<P CLASS=3D"western">While being sane enough to know that building such
a system in my lifetime would be a little out of reach (to put it
mildly), I am just crazy enough to believe it would be a good idea to
get started now! So, I've set out to build just the software portion
of the holodeck. We'll leave all that nasty holographic particle
physics stuff to some future genius. Assuming that characters and
scenes could be rendered by hardware in some appropriate fashion, the
rest is just some interesting software. If we start now, by the time
the physics of the holodeck come to light (literally), we'll have
most of the control software needed. Software will be ahead of
hardware for once!</P>
<P CLASS=3D"western">For several years I collected notes and research
papers, learned the tenets of software development, devoted myself to
providing the best solutions possible in my various jobs, and strived
to gain my own understanding of as many software technologies as I
could. Several things started to happen in software that were very
encouraging. The Internet proved that internetworking could work on a
global scale. SGML's child, HTML, spawned significant interest and
development of one of my favourite technologies: XML. Previously
proprietary advances in hardware, transaction processing, fault
tolerance, clustering and other aspects of large scale systems became
common place. The free software and open source movements became a
commercial reality.</P>
<P CLASS=3D"western">So, what does all this have to do with the
eXtensible Programming System? Everything! There is now sufficient
Open Source software to draw upon, the plan is succinct enough in my
mind to execute, and I could use some interesting work to pass the
idle days.  Consequently, I've started putting the software for XPS
together. Undoubtedly this will be a long process; but, I've always
believed in setting my goals very high so that if I only achieve &frac12;
of what I set out to achieve then I still have accomplished something
worthwhile. Using the concept of the holodeck as a focal point for
the project, has turned out to be quite useful. It is a constant
reminder of the high ideals for the system and causes me to make
design decisions that break through my own pre-conceived notions of
what computing is or ought to be. At the core of the XPS is a
striving to make a quantum leap forward in software technology using
the same old techniques I've always used: incorporating the ideas of
the giants that have gone before me and inventing the rest.</P>
<P CLASS=3D"western">XPS is intended to take software to the next level
by laying the foundation for an extensible machine that will support
the rapid development of whole knowledge domains as yet undiscovered.
It will raise the level of abstraction in programming and make
scalable, network accessible programming available to large
non-technical audiences. In its essence, it will provide the basic
platform from which a system as complex as the holodeck could be
built. It will incorporate every known tenet of computer science and
software technology into a robust platform for the exploration and
development of new ideas. Phew!</P>
<P CLASS=3D"western">So how are we going to do this? I have my own
ideas, but I was counting on you for the real answers!=20
</P>
<H1 CLASS=3D"western">Overview</H1>
<P CLASS=3D"western">The XPS is a large system and I make no apologies
for it. One of its primary goals is to raise the level of abstraction
for programming without sacrificing performance. In its essence, it
provides a virtual machine with a very high level and extensible
instruction set that abstracts away all the details of
multi-processing, multi-threading, synchronization,
internationalization, machine limitations, transaction processing,
network communications, messaging, data storage and retrieval,
mathematical functions, and etc. At its foundation, the system
provides a comprehensive programming language (XPL), the compiler for
that language (XPLC) and a capable virtual machine (XVM). In its
initial incarnation, the system combines some of the best Open Source
software (GNU/Linux, GCC, ACE, Xerces, ICU) to implement these
components. Someday the virtual machine could be implemented in
hardware.=20
</P>
<P CLASS=3D"western">However, all of the above components, lofty as
they may seem, are simply the foundation of XPS. They are the
enablers of two badly needed things in software development:
meta-programming and domain engineering. Meta-programming is the
ability to program in higher level abstractions (preferably without
loss of performance). Domain engineering provides the mechanisms for
developing whole new computing vocabularies that support programming
for specific knowledge domains without resorting to lower level
languages. These two related concepts are at the core of
extensibility and provide the reason for XPS' existence. Even though
the base system (XPL + XPLC + XVM) is quite useful on its own, its
intent is to only provide the common computing base that will enable
extensible programming. To support extensibility, XPS provides an
extension compiler (XPLEC) which processes extensions (written in
XPL) to the system and allows the base system to operate with the
compiled extension. The XPLEC is required to make extension
development easier and to validate that the extension will not
violate the integrity of the system as a whole. Extensions add
fundamental and new operations to the virtual machine. Since XPL
compiles to native machine code, it is used as the fundamental
programming language for extension of the virtual machine. Through
the magic of XML substitution groups, and XSLT, XPL itself can also
be extended to include new information types and primitive operators
(probably coinciding with a virtual machine extension). This
combination of extension capabilities permits entirely new
programming languages to be created using the facilities of XML and
XPS.=20
</P>
<P CLASS=3D"western">A few other things are needed to round out the
picture. It isn't sufficient to simply provide the raw language and
machine that support the system goals. We also need the tools that go
along with programming.Today we use a variety of configuration
management, software versioning, build and compilation tools to deal
with our textual programs. All of these are valuable and the XPS
encapsulates them and provides them as a network accessible service.
Software can be written, developed and tested anywhere, any time and
the XPS will handle all the underlying details of managing the code.
Ultimately, the &ldquo;integrated development environment&rdquo; for
XPS looks like the holodeck too: a virtual world of visual software
components being constructed and assembled through direct
interaction. No more typing!</P>
<P CLASS=3D"western">The XPS provides a comprehensive platform that
handles all the technical aspects of making a very high level
language execute efficiently on an arbitrary machine. It does all
this to support the extension of the virtual machine and the
development of new executable XML based programming languages and
knowledge domains.</P>
<H1 CLASS=3D"western">The XPS Components</H1>
<H2 CLASS=3D"western">XPL: eXtensible Programming Language</H2>
<P CLASS=3D"western">The eXtensible Programming Language incorporates
many of the key features from today's programming disciplines. The
expression structure is reminiscient of Lisp. The object-oriented
features are similar to Java. The algorithmic features are like &ldquo;C&rd=
quo;.
The predicate calculus and fuzzy logic features are similar to
Prolog. However, the syntax of XPL is simply XML and is defined with
an XML Schema Definition. The XPL schema definition was designed from
the ground up with extensibility in mind. It uses substitution groups
heavily to permit incorporation of additions to the definitions of
the basic language components and new primitive operations. While the
language is comprehensive on its own, remember that its only goal is
to provide the &ldquo;computer science&rdquo; parts so that
additional languages can be developed with XPL as the base language.</P>
<P CLASS=3D"western">Becase XPL is simply an XML document, it can be
processed with the full power of XML technologies. Whole new XML
based programming languages can be translated to XPL through the use
of XSLT. XPL programs can be processed for purposes other than
compilation: publishing, visual exploration, semantic analysis,
database storage, etc. The choice to use XML as the syntax language
for XPL makes the assumption that one day all information will be
transmitted in XML or its successor. This isn't an unreasonable
assumption given the prevalence of the world wide web.=20
</P>
<H2 CLASS=3D"western">XPLC: XPL Compiler</H2>
<P CLASS=3D"western">The XPL compiler handles the traditional task of
converting XPL source programs into a format that can be directly
executed by a native machine. However, in the case of the XPS, the
&ldquo;native machine&rdquo; is actually the combination of a virtual
machine (providing the high level instruction set) and native code.
That is, the XPLC converts XPL into sequences of operations on the
virtual machine. Instead of making a &ldquo;portable back end&rdquo;
for the compiler, we've chosen to make a &ldquo;portable instruction
set&rdquo; and ignored the myriad details of native machine code
generation (fortunately, GCC does a wonderful job of that). The XPLC
also handles all the details of semantic analysis of XPL programs.
XPL groups software into packages that belong to some domain. The
compiler produces and saves a symbol table for each package that
successfully compiles. The symbol table can then be referenced
directly by both executing programs and additional compilations. This
makes the execution of the compiler faster (it doesn't have to
recompile all the referent packages) and makes the entire symbol
table available to executing programs.=20
</P>
<H2 CLASS=3D"western">XPLI: XPL Interpreter</H2>
<P CLASS=3D"western">As an intermediate step towards development of the
compiler, an XPL Interpreter was developed. The interpreter parses
and executes XPL code directly. That is, instead of producing a
separately executable program (as the compiler does), the interpreter
invokes operations on the XVM directly as it encounters them while
parsing the XPL software. Interpreted XPL runs much slower than
compiled XPL but it starts executing much sooner because the
expensive compilation step is mostly avoided. In general, developers
will use the interpreter for development, testing, and exploration.
When an XPL package is ready for general consumption, it is then
compiled and made available in its faster compiled form.</P>
<H2 CLASS=3D"western">XVM: eXtensible Virtual Machine</H2>
<P CLASS=3D"western">This is the heart of XPS. All the things that can
be done in an XPL program are executed by the XVM. The XVM implements
the XPS instruction set for a given computing platform. In this
regard, it is much like the Java Virtual Machine but its instruction
set is much larger and extensible. The core instruction set hides all
the &ldquo;nasty details&rdquo; of the inherent functions of the XPS
including memory management, threads, processes, tasks,
synchronization, security, transactions, I/O, etc. All computing
resources are managed by the XVM.</P>
<P CLASS=3D"western">Each instruction in the core instruction set is 16
bits wide, providing for up to 65,536 instructions. To invoke an
extension to the XVM, two core instructions are used to bracket
sequences of extension instructions. There can be 2^32 (about 4
billion) extensions to the XVM, each with its own 16-bit instruction
set. We think of the core instruction set as the &ldquo;default
extension&rdquo; numbered 0.</P>
<H2 CLASS=3D"western">XPLEC: XPL Extension Compiler</H2>
<P CLASS=3D"western">The XPLEC does two things. First, it allows an XPL
package that implements a specific, well-known interface to be used
as an extension to the XVM. Second, it handles the translation of
higher-level programming languages into XPL through XSLT templates
and transformations. In both cases, the XPLEC simplifies the business
of extending the XPS and verifies that each extension is legal and
&ldquo;does no harm&rdquo;.=20
</P>
<H2 CLASS=3D"western">XPSSH: eXtensible Programming System Shell</H2>
<P CLASS=3D"western">This is the program that user's interact with. It
is a shell program that responds to an XML based command language to
get the XPS to do things (such as compile or run a program). Although
it is possible for user's to directly interact with the XPSSH, it is
generally assumed that some other (probably XPS based) program is
handling the actual user interaction and passing on requests to the
XPSSH. XPSSH provides a barrier between what is internal to the XPS
and what is external. It handles all user authentication and
authorization functions before passing requests on to the XPS Daemon
(XPSD).=20
</P>
<H2 CLASS=3D"western">XPSD: eXtensible Programming System Daemon</H2>
<P CLASS=3D"western">This is a meta component of the system. It is a
system services daemon that sits around waiting for an XPSSH instance
to ask it to do something. Users send requests to the XPSD via the
XPSSH which handles all the request validation functions. The XPSD
assumes that when a request is made of it, all the necessary security
precautions have been handled. Consequently, for security reasons,
the XPSD is not a network daemon. It does not accept network
connections. This makes it efficient for the various components of
the system to make low-overhead requests of the daemon in order to
get things done. The functions of the XPSD are broken down into four
sub-services: Build, Develop, Deploy, Execute. The Build service
handles software compilation and automatic dependency checking. The
Develop service handles programmer editing requests, software
versioning, check-in and check-out, configuration management, etc.
The Deploy service handles the deployment of finished software into a
public execution space. The Execute service handles requests to
start, stop, pause, and manage XPL programs. All or none of the
services can be enabled on a given XPS installation depending on the
intended use of the machine.</P>
<H1 CLASS=3D"western">Conclusion</H1>
<P CLASS=3D"western">This paper has provided a cursory and somewhat
tongue-in-cheek overview of the eXtensible Programming System. There
is much more to discover. Use the following URLs to find additional
information:</P>
<UL>
	<LI><P CLASS=3D"western">Rather Verbose Home Page:=20
	<A HREF=3D"www.x10sible.org/">http://www.10sible.org/</A>  or
	<A HREF=3D"../../../../../../../">http://extprosys.sourceforge.net/</A>
		</P>
	<LI><P CLASS=3D"western">Source Forge Project Site:
	<A HREF=3D"http://sourceforge.net/projects/extprosys">http://sourceforge.n=
et/projects/extprosys</A></P>
	<LI><P CLASS=3D"western">Software &amp; Document Download:
	<A HREF=3D"http://sourceforge.net/project/showfiles.php?group_id=3D51164">=
http://sourceforge.net/project/showfiles.php?group_id=3D51164</A>
		</P>
	<LI><P CLASS=3D"western">Subscribe To Our Email List:
	<A HREF=3D"http://sourceforge.net/mail/?group_id=3D51164">http://sourcefor=
ge.net/mail/?group_id=3D51164</A>
		</P>
</UL>
<P CLASS=3D"western">You've have had your appetite whetted, now come
and help us build software's future!</P>
</BODY>
</HTML>