Re: [Emc-developers] MatPLC and EMC, revisited

[<jmk...@at...>]

Fred Proctor wrote:
> Hi All,
> 
> Ensuring consistency of reading and writing shared resources is a huge 
> problem. In our case the resources can either be communication buffer 
> messages or I/O points. The hardware can help somewhat with test-and-set 
> or read-modify-write instructions but these only work on small values 
> (register-length) and not big structures. 

An issue that I have oh-so-cleverly sidestepped by defining the HAL to
work only with individual I/O points.  <grin>  There's not a structure
to be found in my HAL.  Of course that doesn't help the higher level 
parts of EMC, where more complex data structures are needed.  My plan
is for the HAL to come into play just below the position interpolator
and above the PID loop.  I'd like it to incorporate the interpolator 
too, but I don't know enough yet about the interface between the 
interpolator and planner to say if that will work.  In any case, the
first attempt will start even lower, below the PID loop.

> For larger structures something else is necessary. Semaphores are the 
> usual choice but these require OS participation. At one point in RT 
> Linux history semaphores between RT and non-RT tasks weren't available. 
> Even if they are, problems with priority inversion occur. Commercial 
> RTOSes like VxWorks use priority inheritance so that non-RT tasks are 
> bumped up to RT priority if an RT tasks wants a resource the non-RT task 
> is using. The typical use here is with communication buffers, where a 
> non-RT task takes the comm buffer semaphore, copies in the data 
> structure that it had been leisurely modifying earlier, then releases 
> it. For a large structure the RT task will be delayed by the amount of 
> time the non-RT task takes to finish the copy.

Plus the time for a couple of task switches, from RT to the non-RT
task, then back again after the copy.  For some tasks, like a 
software step/direction generator, the task time is already small
compared to the context switch time.  I certainly don't want to add
a couple more task switches to the worst case execution of my fastest
ISR.  For what I want to do, I don't consider priority inheritance
a viable solution, even if it was available in the OS's we use.

> In the EMC I used a different approach, head/tail counts, that does not 
> cause priority inversion but may cause another well-known problem, 
> indefinite postponement. The head/tail technique works like this:
> 
> snip very clear explanation...
> 
> Applying the head/tail technique to the driver-based Hardware 
> Abstraction Layer (HAL) proposed by John Kasunich, here's what
> I think we'd end up with:

Fred, your model is drastically different than what I had in mind.
I'll point out the differences below, then we can debate pros and
cons of each model ;-)

> 1. a driver, or part of a driver, is responsible for each I/O point. I 
> would use the rule of thumb that you have a single driver for all the 
> I/O associated with an interrupt source. The timer can be the interrupt 
> source for boards that don't have their own.

A driver is associated with a piece of hardware.  One for a Servo-to-Go
board, a different one for a parallel port, another one for a PPMC board
set.  If you want to use PPMC for axis control and parallel ports for 
toolchanger I/O, or even PPMC for two axes and STG for the third, no
problem.  Just load both drivers.
 
> 2. The HAL is a shared memory structure that mimics the hardware I/O. 
> It's large and contains many items that aren't atomically 
> readable/writeable. The HAL is partitioned into segments, each bracketed 
> with a head/tail count. There's a segment containing all the inputs 
> associated with an interrupt source, one for all the outputs, and so on 
> for all the I/O.

The HAL is similar to (or is) the MatPLC global memory map.  It consists
of a collection of I/O points, each of which _is_ atomically readable
and writeable.  Each point has an owner, the module or driver that is 
allowed to write it.  The points are named based on their function, _not_
their hardware implementation.  In other words, there are points called
Axis1_VelCmd, Axis2_HiLimit, Axis3_PosFb, etc., not points called
ParPortPin2, STG_AnalogOut3, etc.

Each driver, upon init, reads it own section of the ini file and finds
out which functional point is associated with each of its physical points.
That is where the abstraction takes place.  Note that the motion modules
_always_ think they are dealing with a servo motor, and always write to
Axisn_VelCmd.  If you are using software step/direction, it is implemented
in a driver that reads the VelCmd point at the servo rate and uses a much
faster ISR to generate the step pulses.  The rest of EMC doesn't even
know that the driver is making step pulses.
 
> 3. When the interrupt triggers, the driver checks the head/tail pair of 
> the output segment for a match. If they don't match it ignores the 
> outputs. If they match it writes all the outputs. It then increments the 
> input head count, reads all the inputs into the HAL, and sets the tail 
> to the head.

When an interrupt occurs that activates a driver, the driver reads inputs
from the hardware and atomically writes each individual point to the point
map.  It then atomically reads each output point from the map and writes
it to the hardware.  Note that this method would only be used by drivers
that _require_ a dedicated and possibly asynchronous interrupt.  Most
drivers would export callback functions at init that would be linked into
the servo thread, rather than running as asynchronous interrupts.

In the normal case, the timer interrupt would go off when it's time to 
do a servo update.  First it would call the driver input callback(s),
then it would call the interpolator module, which would write new 
position commands into the map at points called Axisn_PosCmd.  Next it 
would call the PID module, which would read Axisn_PosCmd and Axisn_PosFb
and write to Axisn_VelCmd, then finally the driver output callback(s) 
which take Axisn_VelCmd and turn it into something physical.  All these
calls are made by a single realtime thread, working from a list of
callbacks that are registered as each module (interpolator, PID loop,
I/O drivers, etc.) is loaded and initialized.

Meanwhile, in the background (either lower priority RT or user space),
the PLC task and/or a sequential task like bridgeportio would be doing
it's thing, reading and writing points in the same global map.  Those
tasks can access any point, even if ones normally used by the motion 
module.  I/O devices not updated by the servo loop callbacks would be
updated either by callbacks from the lower priority task(s) or by their
own interrupts.

> 4. If an output writer is interrupted, its values will be ignored until 
> its copying completes. If an input reader is interrupted, it will detect 
> a head/tail mismatch and coast on the previous values.
> 
> Dealing with the head/tail pair is a pain. 

But a neccessary pain when dealing with structs and other non-atomic
things.  Which leads me to a thought that came out in a chat with Ray,
Paul, and myself on Sunday.  I think it is a key concept to grasp in
order to understand where I want to go with the HAL, and it has to do
with dividing the program into high level and low level domains:

I see the EMC, and many other realtime programs, as having two domains.
The high level domain is characterized by complex program flow, and 
command or message oriented data.  In EMC, this would include the GUI,
the g-code interpolator, and the trajectory planner.  NML and other
messaging architectures are natural for that domain.  On the other hand,
the low level domain is characterized by fixed and periodic program flow,
and data that is more state oriented.

By state oriented, I mean that a set of variables represents the state
of the system, and to a greater or lesser extent, it can almost be viewed
as a continuous time or analog system.  Software PID loops, for example,
are a sampled model of what is a heart a continuous time function.  They
run the same code every time, reading the state of their inputs and 
modifying the state of their outputs.  Likewise, ladder logic is a 
sampled model of real relay logic.  Each "point" in my global map 
represents a "signal" in a continuous time system.  (Warning - I'm using
the term "signal" as a hardware guy would, to describe something flowing
from a source to one or more loads, not in the C context where it is a 
type of interrupt sent from one process to another process.)  The
function of the HAL is to serve as a patchboard, routing these 
"signals" between functional blocks and I/O devices as needed.

There must obviously be an interface between the high level world of 
messages and the low level world of signals.  For example, a message
might be sent, saying "start the spindle at 3000 RPM", and expecting
a response of "OK, spindle is turning at 3000 RPM".  I would expect
that the message handler would take that message, and set the "signal"
or MatPLC point Spindle_Run to true, and the point Spindle_SpdCmd to
3000, then wait for the point Spindle_AtSpd to be true, before sending
the response.

Down in the state domain, the implementation depends on the machine.
In the simplest case, Spindle_Run would be mapped directly to the 
spindle start bit of an output port, and a ladder rung would set 
Spindle_AtSpd true 1 second after Spindle_Run goes true.  Or perhaps
an input bit connected to an aux contact of the spindle motor relay
would drive Spindle_AtSpd.  A machine with a VFD on the spindle might
be more complex - Spindle_Run would start the VFD, and Spindle_SpdCmd
would be sent through a ramp block to an analog output and then to
the speed reference input of the VFD.  A comparator block would set
SpindleAtSpd true when the ramp output is equal to Spindle_SpdCmd, or
perhaps it would actually compare real speed feedback to the command.

The basic idea here is to enable the low level, state oriented part
of EMC to be customized by systems integrators and others who may not
know C programming, but are very familiar with PLCs, ladder logic, and
such.  MatPLC already has a fairly rich set of modules, such as PID 
loops, ramps, ladder rungs, and so on, but is not currently able to
run in realtime.  EMC has the ability to parse g-code and plan a motion
trajectory, and also can execute that trajectory in realtime, but is
limited in how it deals with different machines and I/O devices at the
lowest level.  I want to combine the strengths of the two projects.

Oops, got on the soapbox again, I gotta stop doing that....

Comments?

John Kasunich

PS:  MatPLC folks, I started cross-posting this thread because I wanted
input about using MatPLC in EMC.  This most recent message is much more
EMC focused - if we're off-topic, speak up and we'll keep it on our list
only.  But I think it might provoke some interesting conversation, so
I'll keep cross posting until told to stop.