Re: [Embedlets-dev] Interested in sharing a J2EE perspective

[Gregg G. W. <gr...@sk...>]

As I've discussed here before Cyte Technologies has been building device to 
boardroom systems since 1997.  The issues that Ted brings up are very real.  
It is important to understand these points and implement solutions, 
communications and strategy messages that work from the bottom of these 
organizations to the top.

Some key issues related to the scalability Ted discussed revolve around making 
the correct decisions about responsibilities at all levels to minimize 
'network' traffic and administrative load.

o	If a device is able to detect a particular status, it should be
	able to throttle the reporting of that status based on configuration
	and importantance.

o	Polling is not a generally scaleable strategy.  At the upper levels of
	the system, you must use report by exception for status data.  A timely
	healthcheck must be included to allow you to determine that the device
	is still operating, and to refresh the world with the 'current' status
	periodically.  Think of the MPEG-4 I-Frame mechanism which allows video
	displays to randomly be inserted into the video stream without having to
	see the very first frame in order to understand what the current display
	should be.

o	Data that must be delivered should use an appropriate transactional
	mechanism that allows the data brokers to fail, and for the field devices
	to know what data is still undelivered.  This helps to eliminate large
	scale data recovery which can be impossible to administrate as the number
	of devices reporting data grows large.

o	In line with the previous issue, administration of the field devices
	must be automated as much as possible.  Any variant data/configuration
	must have an enterprise wide storage location that can be reached from
	all locations in the system.  Replacement devices can then be
	reconfigured for their installed location in an automated way that
	guarantees they are operating with the correct software and configuration.

Our system was started before J2EE existed and we've continued to use it 
because it is well tuned and instrumented for this type of application.  One 
of our current applications is doing approximately 120,000 transactions per 
hour against 3 different databases, as well as delivering 40,000+ data and 
status valued events into our system monitoring and control application.  
There are over 1500 end devices being polled by approximately 150 relay 
devices that do the status monitoring and reporting by exception and required, 
timely data reporting.  The devices are delivering data at a rate of 
approximately 70 packets per second.  There are 2 machines receiving this 
data, each a hot standby for the other.  These machines are running linux, and 
have load averages < 2.

Our data routing software architecture is a multi-tiered system with decoupled 
inbound and outbound data delivery using extensive multi-threading.  The 
primary latency that we deal with is database transactions.  We typically run 
between 5 and 10 threads for each type of data delivery into the databases.  
This is necessary to help make sure that we keep up with the inbound data 
rates.

Having a well tuned, well maintained enterprise database system is a real 
necessity.  One of our customers is an aluminum melting enterprise.  We are 
building furnace control systems that include data gathering and remote 
furnace control capabilities.  They have little experienced IT infrastructure. 
 So, we have to do more hand holding and provide less technical and more do-it 
documentation to make it possible for them to successfully utilize the power 
of the systems that we are deploying.

So, everthing that Ted is discussing is not a single persons experience.  
These are many of the primary issues that everyone needs to be aware of.  I'm 
really interested in hearing about other experiences that the J2EE folks have 
had, and in particular are there common themes present in other types of 
integration applications.

-----
gr...@cy...  (Cyte Technologies Inc)