[mcnet4l-devel] Re: responder step 4

[Daniel S. <st...@in...>]

hi mark et al.

On Wed, 2001-12-12 at 16:42, Daniel Stodden wrote:

> You are not confused at all.  You perfectly understand the difficulties
> of using an out-of-band interrupt in 2.14.  The committee pondered this
> at length.  The choice was an awkward ISR implementation, an ad hoc
> driver for each class of 2.14 hardware that potentially would interfer
> with other uses of the same hardware node, or a prohibition of
> out-of-band interrupts.  We chose awkward. Just how awkward depends on
> the rest of the hardware implementation. If the hardware funnels many
> interrupts through a particular interrupt port, then locating the
> initiator node may take some time, although the implementation on the SN
> can mitigate this by limiting the number of outstanding connection
> request that it accepts.  And, you are correct, it is possible to lose
> the use of the interrupt system completely if the PN messes up the
> management info.  Of course, the PN can normally just scribble all of
> the SN's memory to much more spectacular effect 8-).  I suppose we could
> have passed some more information in the initiator CCB - for example, we
> might say that the local management means are filled out with the
> management means that would be given if the responder were to ask for a
> bus interrupt.

grmpf. fine, i've had a proposal exactly like this in mind while i wrote
the original email.
guess i should write more extensive problem descriptions or carsten will
put _your_ name on my thesis paper.

  This would probably entail an additional phase to the
> connection process to excahnge the speculative management means followed
> by the actual management means.  I will keep that in mind if we get to a
> second revision of the spec. 
> 
> SNs can avoid this problem by using memory polled alerts or by having
> hardware which allows inband interrupts.

well, without hardware support, at least the SAM ends up
polled on system host PNs. definitely -- unless above management
extension would become part of the next revision.

> As for explaining it, well, the spec is already much more expository
> than normal for such specifications.  We needed to leave some material
> for your thesis!  To say nothing of my consulting business.

i agree. on the other hand, there are already enough "sidenotes" in 
being actually more precise than one might expect.

i partially find this pretty disturbing. after all, the specification
has been built around PICMG2.x, while leaving even the most basic
property -- host vs. peripheral -- more or less undiscussed.
from that standpoint, you guys should then rather have tried to avoid
terms like "CompactPCI" or even "PCI" in general...  :)

> Some hints for the implementation of such an ISR - 
> The SN may know generally which slots have outstanding requests, and
> hence can limit the potential interrupt routes/sources that need to
> execute the "speculative" isr code.
> 
> Using the flush before alert property ensures that the CCB will have a
> state change prior to the arrival of the actual interrupt.  This makes
> the search for potential interrupt sources relatively quick, especially
> if the number of outstanding requests has been limited by the SN.
> 
> The good news is that this situation is transient, and the SN knows
> whether it needs to perform the additional search or not, based upon
> whether the SN has outstanding connection responses.  The problem is
> that there needs to be a sort of general interrupt handler trailer that
> gets conditionally called if the interrupt is not from any known
> source.  Perhaps the isr routine can be added to the line where the
> interrupt will occur as needed, and generally not be part of the isr
> chain when not needed.
> 
> That is - there is a general_214_search_for_new_interrupt_source routine
> that is registerd as an isr for any line on which a bus 214 interrupt
> might occur.  This registration is done as part of the connection
> response prior to entering responder step 4.  Then, an unacknowledge
> interrupt on that line will enter the general search handler.  There
> will be a limited number of potential 214 sources for that interrupt,
> which the general handler will search.  If it finds new management, it
> will set that up and register a new handler, and if there are no more
> pending responses, remove itself from the isr chain.  A bit tricky,
> perhaps, but not a whole lot of overhead, and not intrusive to the rest
> of the interrupt system.  Just watch the SMP stuff.  

(this last sentence i did not understand)

Note that the
> general handler never acks the interrupt, but rather adds the real
> interrupt handler to the list to actually perform the ack after the
> general handler returns.

neat idea. especially since it would allow to shift the whole 214 irq
dispatch chain over to the interrupt system without need to keep
separate (connection-oriented) queues in a driver.

however, for a few subtle reasons, it just won't work :P
i don't know about other operating systems, but the linux interrupt
system ist just not prepared to mess around with the isr chains
from within irq context.
the expectations for conventional device drivers are simple:
- probe for the device
- set it up
- register the interrupt handler
- run for a week, the unregister the handler on driver shutdown.
and do that in process context, e.g. insmod/rmmod.
for 99.9% of all possible hardware this is perfectly o.k. 
for 2.14 obviously not :)

that's what the kernel folks expect to happen, and as a result,
request_irq() is doing GFP_KERNEL memory allocations for it's irqaction
descriptor and therefore will crash when called from a bottom half, let
alone from another isr.
free_irq will hang if you try to release the line from the isr itself.

look into arch/i386/kernel/irq.c. in smp builds, free_irq will block
if called from the isr it's going to unregister (is that what you meant
with "watch smp" above?)
so switching isrs directly from within is simply not possible.
at least not without just rewriting the interrupt system of any
architecture we are going to run on B)

getting process context is not too much of a problem during "normal"
operations. the mcdev interface has been changed in the meanwhile to
allow for a connect() operation, with the very first bus reference (i.e.
RL or SAM) as a parameter. while the connection setup state machines are
still assuming to run from bottom halves, calls to connect() are run
from keventd. this allows the host device to figure out which device the
address refers to and register an isr on the first connection made.
nowadays, i'm thinking whether it would be a better idea to create plain
kernel threads for connection setup, but right now it's just step4 which
gives me a headache.

fwiw, as far as i can tell from here, 2.14 bus interrupt dispatch will
end up with chains maintained by the respective device driver.

i've mostly been asking in order to make sure not to misunderstand
things. obviously i didn't. good to know. thanks so far :)

btw, the 21554 code should be functional (assuming i didn't mess up cvs
contents over the last few days). if you find the time and interest to
try it out let me know. there's still no discovery support and not much
comments in the tree, so i would write a short howto about
"howto-get-it-going" again.
it's now about three month since my last preview release. waaay too much
time. should have done so with the drawbridge drivers. maybe even
before. i thought the hostboard stuff would turn out to be much simpler.
funny :) 

thanks,
dns

-- 
__________________________________________________________________
mailto: st...@in...