Re: [mcnet4l-devel] responder step 4

[Mark H. <mh...@mv...>]

Daniel Stodden wrote:
> 
> mark,
> 
> i'm close to finishing 2.14 device drivers for both
> system host boards as well as 21554 based peripherals.
> 
> regarding the host device, i think i've stumbled over
> a general problem with the connection setup scheme,
> related to out-of-band control alert managements on the receiver side.
> 
> i just did not notice this before, since all the in-band
> alerts, e.g. 21554 doorbell interrupts, on peripherals usually do fine
> without management descriptors provided by the remote side.
> 
> consider a peripheral (21554) initiating a connection to a
> node residing on the system host:
> 
> the initiator would deliver alert descriptors based on 21554 doorbell
> interrupts. the responder on the system host would request bus
> interrupts. here, the responder needs to rely completely on management
> descriptors delivered by the initiator.
> 
> the weirdness starts in initiator step4/responder step4:
> here, the responder gets an interrupt from a device it effectively has
> not yet seen managements for -- the managements are provded within the
> same step in which the interrupt occurs.
> so a driver would have to
> - parse the managements
> - apply the identification mechanism. see if it succeeds.
> - apply the clear means.
> 
> all this - basically the whole "responder step4" procedure - will need
> to take place directly within the interrupt handling routine. not
> impossible, but pretty much work for an irq service, i think.
> 
> interrupts need to be assumed as shared with other devices (both
> mcdevices as well as non-mc). so, this has to happen based on an
> _assumption_ the interrupt stems from a respective connection.
> 
> if the remote side failed to deliver proper management mechanisms, the
> whole responding system gets more or less trashed. no way to clear the
> interrupt execept disabling the entire irq line.
> 
> i don't really feel comfortable with this scheme.
> has the 2.14 committee been aware of this? maybe the spec should note
> some of the implications regarding dependencies on alert managements?
> 
> confused again :),
> dns
> 
> --
> __________________________________________________________________
> mailto: st...@in...
> 
> _______________________________________________
> mcnet4l-devel mailing list
> mcn...@li...
> https://lists.sourceforge.net/lists/listinfo/mcnet4l-devel

Daniel,

You are not confused at all.  You perfectly understand the difficulties
of using an out-of-band interrupt in 2.14.  The committee pondered this
at length.  The choice was an awkward ISR implementation, an ad hoc
driver for each class of 2.14 hardware that potentially would interfer
with other uses of the same hardware node, or a prohibition of
out-of-band interrupts.  We chose awkward. Just how awkward depends on
the rest of the hardware implementation. If the hardware funnels many
interrupts through a particular interrupt port, then locating the
initiator node may take some time, although the implementation on the SN
can mitigate this by limiting the number of outstanding connection
request that it accepts.  And, you are correct, it is possible to lose
the use of the interrupt system completely if the PN messes up the
management info.  Of course, the PN can normally just scribble all of
the SN's memory to much more spectacular effect 8-).  I suppose we could
have passed some more information in the initiator CCB - for example, we
might say that the local management means are filled out with the
management means that would be given if the responder were to ask for a
bus interrupt.  This would probably entail an additional phase to the
connection process to excahnge the speculative management means followed
by the actual management means.  I will keep that in mind if we get to a
second revision of the spec. 

SNs can avoid this problem by using memory polled alerts or by having
hardware which allows inband interrupts.

As for explaining it, well, the spec is already much more expository
than normal for such specifications.  We needed to leave some material
for your thesis!  To say nothing of my consulting business.

Some hints for the implementation of such an ISR - 
The SN may know generally which slots have outstanding requests, and
hence can limit the potential interrupt routes/sources that need to
execute the "speculative" isr code.

Using the flush before alert property ensures that the CCB will have a
state change prior to the arrival of the actual interrupt.  This makes
the search for potential interrupt sources relatively quick, especially
if the number of outstanding requests has been limited by the SN.

The good news is that this situation is transient, and the SN knows
whether it needs to perform the additional search or not, based upon
whether the SN has outstanding connection responses.  The problem is
that there needs to be a sort of general interrupt handler trailer that
gets conditionally called if the interrupt is not from any known
source.  Perhaps the isr routine can be added to the line where the
interrupt will occur as needed, and generally not be part of the isr
chain when not needed.

That is - there is a general_214_search_for_new_interrupt_source routine
that is registerd as an isr for any line on which a bus 214 interrupt
might occur.  This registration is done as part of the connection
response prior to entering responder step 4.  Then, an unacknowledge
interrupt on that line will enter the general search handler.  There
will be a limited number of potential 214 sources for that interrupt,
which the general handler will search.  If it finds new management, it
will set that up and register a new handler, and if there are no more
pending responses, remove itself from the isr chain.  A bit tricky,
perhaps, but not a whole lot of overhead, and not intrusive to the rest
of the interrupt system.  Just watch the SMP stuff.  Note that the
general handler never acks the interrupt, but rather adds the real
interrupt handler to the list to actually perform the ack after the
general handler returns.

Sorry for the back of the table cloth design, but you get the point.

Regards,
Mark Huth
MontaVista Software