[Embedlets-developer] Re: Design Thoughts (reponses to Ted's points)

[Andrzej J. T. <an...@ch...>]

Ted sed:

> All the design documents are in OpenOffice 1.0's XML file
> formats so they can be managed by a CVS system very easily.

That's very cool.  I installed StarOffice a while back, but haven't used it much. 
This might be a good excuse to do so.

> Embedlet logic can be developed using:
> 
> 1)  Java.  

That's probably a given.... <grins>

> 2)  PICBASIC.  For people who are not skilled Java developers there is no
> reason why a PICBASIC interpreter Embedlet could not be created and then
> people who know how to program in PICBASIC can easily develop their own
> Embedlet logic.  If Embedlets are capable of being graphically wired
> together using a wiring tool then these people will not even necessarily
> be aware that they are using Java at all.

I think that anyone that can figure out how to use PICBASIC should be able to 
use Java in a simple scripting mode.  But maybe I am mistaken on that....and 
regardless, you are correct that an interpreting embedlet could be written quite 
easily.  Better still, would be a code generator that took PICBASIC and 
translated it into Java, which would then be compiled.  Better performance that 
way on limited devices.

> 3)  LADDER LOGIC.  If Embedlet logic can be developed in PICBASIC why not
> allow it to also be graphically developed using LADDER LOGIC?

It could...but I don't think that should be the initial focus of our efforts. Once we 
have a few Outpost/Embedlet containers running using Java, then we can see 
what makes sense in the way of additional language/scripting support.

> The deeper that I dig into developing a solution to the Elevator challenge
> the more I am convinced that Embedlets are capable of elegantly
> accommodating an extremely diverse set of needs.

Embedlets are just a container design for limited devices, with a full 
programming language included (Java in this case).  With that flexibility you 
can implement anything that is Turing Complete.  The question is not can 
we....but should we, and if so, when?

> most people seem to have forgotten that
> JavaBeans were originally designed as a general purpose software component
> technology capable of supporting a large software component market for
> people that wanted to graphically wire applications together.  Beyond
> this, with EJBs and the enterprise market taking off like it did there
> were little resources or desire to pursue the original JavaBeans vision.

I would add that there has not been any substantial market demand for such 
graphical programming/assembly tools.  On any platform (not just Java).  Even 
with COM components and VB assembly, the issue you come down to is that 
custom application logic has to be coded in a programming language (and I 
don't see that changing in the near term....Java/C/C++/Perl/Basic aren't going 
away for a long time).  And as soon as you have to write code, the benefits of 
switching to a graphical tool for part of your work lessens substantially.

> The new BeanBuilder tool is open source and I would like to use it as the
> foundation for the initial proof-of-concept Embedlet Wiring tool.

I have no issue with that.  So long as the GUI tool creates the underlying 
Embedlet XML config file that can be then pumped through the Embedlet 
configurator/builder.  But somehow I think this will be a philosophical sticking 
point....

> We are in complete agreement here.  The Embedlets themselves will often be
> very complex and so they will probably be developed in Java by skilled
> Java programmers.  The graphical wiring capabilities are just for allowing
> non-programmer application assemblers to assemble applications using
> standard software components (Embedlets).

If all you are doing is simple wiring, then having the GUI generate an XML 
Config file that specifies the "glue" is not that hard to do.

> I agree that the container could offer standard services but I think that
> the services should be pluggable so that if one wanted to put a minimal
> service Embedlet Container in a uVM it could be easily done.

Absolutely....as many services as possible need to be optional.  The ones that 
would be mandatory would include Event Management, Component Lifecycle 
Management and Scheduling (more on the scheduling issue later).  Optional 
ones could be: Logging, Management (JMX), and others.

> I think that we will need to discuss this point further.  I think that if
> a non-programmer application assembler is developing an Embedlet based
> application then they should be able to do so by directly wiring the
> application together using a graphic wiring tool (which is a technique
> that has been proven to work well by JavaStudio and the various versions
> of the BeanBuilder).

I think this wiring should be specified by an XML config file.  Which in turn 
could be generated by a graphical tool for those that are so inclined.

I'm still nervous about the graphical construction approach, since it has never 
succeeded in the marketplace, whereas XML-based configuration capability 
has.

> I do not see why we can not provide both mechanisms especially since they
> seem to be complementary.

There is no reason we can't do both....

> So far the experience I have had with developing a solution to the
> elevator problem indicates that quite complex applications can be
> assembled by graphically configuring components and wiring them together. 
> I hope to have my solution available for critiquing within 2 weeks so that
> we can see if I am missing something critical.

The XML Config specification, if it contains a comprehensive, standalone 
description of the config'ed app, would probably still be rather complex.  That 
is not a really big deal in my mind, since the XML would only be processed on 
a development workstation (eg. PC) and not on the embedded device itself.

> The route that the wiring tool is taking is to allow the application to be
> graphically wired together and then the application's complete object
> graph is saved using the LTP API and then also deserialized into the
> embedded system.  I have already successfully deserialized small
> applications into a TINI as a proof of concept (it worked well) and I
> think that this technique will work with the uVM too.

I prefer a code-generated approach (for the most constrained devices) and a 
dynamic config approach (for more capable ones).  My gut feel is that it will 
provide more flexibility and performance that way, rather than using serialized 
instances.  The bean assembly approach (with it's serialized persistence 
metaphor) works well in scenarios where you have fewer classes than 
object....that is, when you instantiate many objects of a particular class.  This 
is common with apps that are UI based, where you instantiate visible widgets 
like text edits, scroll bars, labels, etc.  I don't see much value to that approach 
when you there aren't as many objects and the classes do most of the work 
(as would be the case with embedlets).  The level of complexity is reduced 
substantially, since there are not that many objects to instantiate, and the 
server process can do that based on the config info (rather than re-consititute 
a serialized instance).

> I think that we should definitively prove that one Embedlet specification
> is incapable of spanning the full spectrum of target environments before
> falling back on multiple specifications.  I currently think that one
> specification will be sufficient.

I agree.  One spec.  But there may be many different container 
implementations based on that spec.  aJile chips have threads, so why not 
have a container that can leverage that?  uVM may not have threads, so a 
different container may be needed.

> I think that, just like with security, if the graphical wiring capability
> is not built in from day one that it can not be successfully be tacked on
> afterwards.  In order to accommodate graphic wiring, Embedlets simply need
> to permit their properties to be configured using setters and getters
> (achieved mostly through naming conventions) and they need to allow for
> the registering, unregistering and handling of events.  They should suffer
> no more than normal JavaBeans have by being able to be using in a graphic
> wiring tool.

The graphical tool could just as easily set the property values in the XML 
Config file rather than using bean-based introspection and the like.  So that's 
not a big issue.  I think we can also build a container where the actual 
embedlet code can avoid registration/unregistration for events by having that 
registration handled by the container on the components behalf (we might 
want a way to unregister dynamically, though I think this will not be needed 
much since embedlets will typically listen to a small number of events all the 
time, and that might be better handled with lifecycle events).  In any case, the 
simple cases should probably have such "plumbing" issues handled 
automatically based on an XML descriptor (as I had proposed), with a dynamic 
API available for the odd case that might be needed.

> With JAPL, Embedlets can handle all possible 'peripherals' from simple
> devices to the most complex backend system.  Again, this is shown in the
> following simple Embedlet application (notice that it is hooked up to a
> backend enterprise system):

I don't think JAPL is necessarily a good way to abstract communication to the 
host systems (a specialized embedlet might make more sense for that).  And 
having a standard API for device access is a good thing, but there are some 
scheduling issues if you are running in a container environment.

The focus of control of scheduling (and things like event management across 
multiple embedlets) is the Container.  So an embedlet should not grab the 
processor and keep it.  If the platform supports threads, then this is easy to 
implement (eg. PC's, aJile boards).  The problem is if you don't have threads.  
Then the embedlets need to use a co-operative processing approach where 
they relinquish control back to the container rather quickly...probably just 
responding to events as they get them.

What if an embedlet needs to regularly monitor a bit on a physical device?  
Sure JAPL lets you use a nice abstracted API to get the bit.  But without 
threads, how do you construct an Embedlet that can do periodic tests using 
the JAPL API call?  It can't just do a delay loop with a test, since then the 
container and other embedlets won't get any processor time.  One way is to 
have a PeriodicTimerEvent that Embedlets can subscribe to receive.  And 
each time it receives such a timer fired event it does one test using JAPL and 
then does something useful (maybe posting a different event).  This way you 
can have multiple embedlets running, without needing threads.

I don't think that it would make sense to embed these kinds of "polling" 
semantics into the JAPL API.  Rather, we could have some standard "off-the-
shelf" embedlets (and rather simple ones) that implement such semantics, 
basically talking to a device as required (whether polled, interrupt driven, 
threaded, etc.) and then translating the device data into a suitable 
EmbedletEvent that gets propagated further.  They would conceptually be 
"adapter" embedlets that translated between the hardware bits and the 
Embedlet Event paradigm.

> My primary goal is not to get enterprise developers more comfortable with
> embedded development.  My goal is to allow the semi-technical IT personnel
> which are present in most companies to be able to graphically wire
> together Embedded systems and attach them to backend enterprise systems
> with minimal help from the the enterprise developers.

This assumes that they won't need custom application logic on the embedded 
device.  That assumption might not hold very well in the real world, unless the 
scope of what embedlets do is rather reduced.

> My thought was that the $25/hour IT personnel should not have to bother
> the $100/hour enterprise personnel more often than was absolutely
> necessary in order to attach the eventually large number of embedded
> systems that companies will eventually need to be hooking to their
> enterprise systems.

If you are interfacing with enterprise systems, the enterprise guys will be 
involved.  No way they will allow anyone to plug just anything into the large 
back end systems.  Initially the end to end solution would have to be 
architected, which would involve everyone. Embedlets would be written.  After 
that, deploying more of the same types of controllers might be allowed, but 
initially I think it will involve everyone.  At best, the server side people will insist 
on massive integration testing.

> In my scheme, the enterprise developers are primarily used as a 'back
> door' method for marketing the Outpost idea to companies.  Even though
> some of them might catch the embedded systems bug, I think that most of
> them will not have the necessary background for interfacing embedded
> systems to their company's processes.

Maybe not....but they will be involved, and actively so, in any new 
implementation that interfaces with their back ends.  In fact, they would 
probably have to write new servlets (exposed as web services perhaps) to 
facilitate the integration, which means both side have to talk a lot.

> Without graphic wiring capabilities Embedlets are not usable by my pet
> target user base.

I don't think we should be targeting non-IT end-users for Embedlets.  The 
reason IT exists is to translate user requirements into a logical 
flow...something most users are not great at.  Graphical tools don't really help 
that problem.  And plant-level IT folks are definitely capable of using XML 
based configurations and writing some simple logic.

> I spent a significant amount of time digging into the dynamic proxy API
> and it will only run on JDK 1.3 and up APIs.  Numerous classes in the API
> were enhanced to accommodate dynamic proxies (like the serialization API)
> and almost no embedded Java system are going to have these capabilities in
> the near future.

I was thinking of PC's....there is a lot of applicability for using stripped down, 
hardened PC's on the plant floor as both controllers or concentrators (Level 1 
or 2 as was defined in the SoftPLC article).  A minimal PC with linux and a 
JVM would make a great Outpost platform for many applications.  And in that 
case, why not have a Container implementation that allows for more dynamic 
operations and leverage things like dynamic proxies, threads and more?

My only point is that the Outpost specifications should not preclude such 
container implementations for more capable platforms.  But it should not 
require such advanced capabilities either.

> Since all Embedlets that need to receive events implement one interface
> there is no need for dynamic event proxies.  

There may be value in allowing an embedlet to receive different events 
through different methods.  It potentially gets rid of long Select/Case chains in 
the Embedlet logic, letting the container do the "routing" for the embedlet.

> Simply have an Embedlet
> register itself as an event listener for a specific type of EmbedletEvent
> on any given target Embedlet and thats it. 

Better still, have the container register the embedlet as part of the initialization 
process, if the config says that the embedlet wants to receive certain event 
types.

> I read through your event mechanism description and, believe it or not, it
> is almost identical to the event mechanism that was developed for JavaOS. 
> I think the fact that you independently came up with an event mechanism
> that is very close to the event mechanism that the pricipal designer of
> Apple's PCI device driver architecture, Sun and IBM arrived at after years
> of effort is significant.

It's also very similar to best practices in the Messaging world (JMS) and event-
driven windowing systems (which is where it all began).  This stuff has a large 
body of "best practices" out there.

> Where our current approaches differ is 1) direct event registering
> vs.using an event broker and  

There would still be an event broker (the container itself) in either scenario.  
It's just that letting the container (rather than the component) handle the 
registration might have some benefits, per my earlier comments.

2) having multiple interfaces for
> registering event listeners on a event producer vs. having one generic
> event registration interface for all producers.

How about no interfaces (at least not mandatory ones) for the registration, and 
let the container handle that.

>  I am
> not sure how dynamic a typical embedded system needs to be.

As soon as you have more than one embedlet running, you get into the 
scheduling issues I mentioned.  Which implies the need for a Scheduler and 
Event Manager (broker)....which I see as the responsibility of the Container.

> I think that some more discussion on this topic and some experimenting
> would be sufficient to clarify these two issues for me.

Yup...

> Is there any chance that I could persuade you to obtain a copy of the
> JavaOS book (Inside the JavaOS Operating System, Saulpaugh and Mirho,
> Addison-Wesley, ISBN 0-201-18393-5, available from Amazon) so that we can
> discuss the merits of an event broker approach using a working and proven
> example as a starting point?

I don't think my schedule will allow it....and I don't think it's necessary.  Event 
and Message-based systems are something I know a lot about.

> From the research I have done on JavaBeans it appears to me that the two
> things that killed its potential for creating a large component market was
> 1) the lack of a robust method for long term persistence of object graphs
> and 2) an event mechanism that was not general enough to allow an
> extremely wide range of software components to be easily wired together. 
> Number 1 has already been solved and I think we have an excellent shot at
> solving number 2.

I think #2 will still take some time, and will likely build on the work in the Web Services area to some degree.  The difficulty in wiring a wide range of components is that it invariably requires some custom logic....which is easier to do in a programming language than in a graphical tool.  
> As for EJBs, it seems to me that doing enterprise Java development right
> is so complex that highly skilled Java programmers are absolutely needed
> in order to 'knit' everything together.  

Hence the need for us to keep the embedlet spec and related XML configs as 
simple as possible.  We have an advantage, since most embedded processors 
will not need the same scalability, security, transactional and flexibility 
characteristics that are needed on the enterprise end.

> If even the Java developers can
> not agree on fundamental things like whether Container Managed Persistence
> or Bean Managed Persistence is the best persistence mechanism to use then,
> to me, this is an indicator that EJBs are not yet mature enough to support
> non-programmer application assemblers.

That has nothing to do with component assembly....and everything to do with 
the inflexibility of CMP in real-world situations, versus complex O/R Mapping 
tools typically used for BMP.  The EJB spec is also fundamentally flawed in 
that it mixes together a number of requirements that are orthogonal to each 
other, such as security, transactionality, scalability and persistence.  Note that 
the first three are typically cross-cutting concerns, whereas persistence is 
more localized.

> The Embedlet space is looking like it is significantly simpler than the
> EJB space and the more I work on a solution to the elevator problem the
> more I am convinced that Embedlets have a real shot of being the first
> Java software component technology to create a large and successful
> component market.

The issue of granularity also works against widespread wiring of finer-grained 
components....and we haven't even resolved and deployed solutions to wiring 
of much coarser-grained services (though the web services community is 
working on that).  

Past history with software component markets would lead me to believe that 
we are a ways from that.  If such a market arises, great...but I don't think the 
creation of such a market should be our primary consideration.  

> Over the past few weeks JAPL has been developed to solve the device
> drivers problem.

Where can I get more info on the JAPL API?  Does it resolve the co-operative 
processing of multiple components issue (polling) problem (which I think it 
should not....that should probably be left to adapter Embedlet components 
rather than complicate the JAPL API).

> Could you draw up a model of the same application that uses an alternative
> to JAPL to solve the device driver problem?

It would look just like your diagram with the only exception being that I don't 
think the API should handle the host communications.  It should stick to being 
a VPL API definition only.  The point I am trying to clarify is where does the 
event scheduling happen, which is not specified in the diagram. I think it 
should be in the Sensor and Alarm Embedlets....using an approach similar to 
the "polling" example I mentioned above.

> What do you think about creating a 'Taramina' folder in the dev_docs
> directory of the CVS and placing your design work under version control?

Sure...when I have something ready, but I would rather we set up a generic 
"scratchpad" area for such musings, rather than tie the folders to specific 
names  Most of this is still in napkin diagrams or in my head right now.

> Together the JAPL specification and the Embedlets specification solve so
> many of the embedded space's current problems that, if they succeed at
> all, I think they will succeed in a very big way.  Thankfully, we are
> mostly responsible for developing the open specifications and providing
> open source reference implementations for them.  If these are done 'right'
> then the embedded space should just naturally build to these
> specifications just like the world built to the HTTP/HTML specifications
> when they were first released.

So the theory goes. ;-)

> 1) Embedlets provide platform-independent embedded logic at the binary
> level and  the Embedlet Container provides a virtual embedded system that
> will allow embedded applications to be completely shielded from the
> grueling platform churn and diversity that the embedded space is
> susceptible to.

The container should also provide some core services (and optional ones) so 
as to make embedlet code as simple as possible.

> 4) All the I/O chip manufactures make similar competing chips that are
> incompatible with each other.  For example, there are dozens of UARTS on
> the market and they all need unique device drivers.  Even embedded Java
> systems like the JStamp and TINI use proprietary classes for accessing
> things as simple as an 8 bit I/O port.  JAPL solves this problem for the
> application developer by coming up with one generic interface for each
> chip/peripheral type.  Now, all client code is compiled against one
> generic interface for each device type and then the task of developing a
> device driver which will allow any specific chip variant, like an 8250
> UART, to realize this interface is significantly reduced.  This represents
> the successful JDBC model brought into the embedded systems world.

Absolutely agree.

> 5) It allows embedded systems to be graphically wired together by
> non-programmer IT personnel.

I see this as a "nice to have" in the short term.

> 6) It enables the creation of a large software component market for the
> embedded space.

That is a risky target.  I would be happier just addressing the "pyramid" 
problem of getting an enterprise "wired" so that info flows from the shop floor 
right up to the enterprise systems (from embedded controllers to concentrators 
to server-side enterprise systems) as the initial primary goal.

> 7) Perhaps the most compelling story that these specifications have to
> offer is that they represent what I think may be the *first* wide scope,
> open source community and technology to be targeted toward the embedded
> space.  In an earlier post you were so right when you pointed out that
> almost all the endeavors to date in the embedded space have been extremely
> proprietary.

Yup....that is a key benefit of our approach.

> So, if JAPL and the Embedlet Container specification can actually succeed
> in providing the first open embedded systems platform and community that
> the whole embedded system's space can leverage, then I think it will be
> extremely successful.

It might.....but is still based on Java availability.  I think it's only now become 
feasible to deploy Java on smaller devices (aJile/Systronix leading the way 
here, and uVM coming soon hopefully), and so now we can leverage the 
ubiquity of Java down into the embedded space.

> I think the embedded space is ripe for the introduction of open source and
> I think we are in a position to significantly help with this.

For sure!


Andrzej Jan Taramina
Chaeron Corporation: Enterprise System Solutions
http://www.chaeron.com