mcnet4l-devel

[mcnet4l-devel] Re: responder step 4

[Daniel S. <st...@in...>]

hi mark et al.

On Wed, 2001-12-12 at 16:42, Daniel Stodden wrote:

> You are not confused at all.  You perfectly understand the difficulties
> of using an out-of-band interrupt in 2.14.  The committee pondered this
> at length.  The choice was an awkward ISR implementation, an ad hoc
> driver for each class of 2.14 hardware that potentially would interfer
> with other uses of the same hardware node, or a prohibition of
> out-of-band interrupts.  We chose awkward. Just how awkward depends on
> the rest of the hardware implementation. If the hardware funnels many
> interrupts through a particular interrupt port, then locating the
> initiator node may take some time, although the implementation on the SN
> can mitigate this by limiting the number of outstanding connection
> request that it accepts.  And, you are correct, it is possible to lose
> the use of the interrupt system completely if the PN messes up the
> management info.  Of course, the PN can normally just scribble all of
> the SN's memory to much more spectacular effect 8-).  I suppose we could
> have passed some more information in the initiator CCB - for example, we
> might say that the local management means are filled out with the
> management means that would be given if the responder were to ask for a
> bus interrupt.

grmpf. fine, i've had a proposal exactly like this in mind while i wrote
the original email.
guess i should write more extensive problem descriptions or carsten will
put _your_ name on my thesis paper.

  This would probably entail an additional phase to the
> connection process to excahnge the speculative management means followed
> by the actual management means.  I will keep that in mind if we get to a
> second revision of the spec. 
> 
> SNs can avoid this problem by using memory polled alerts or by having
> hardware which allows inband interrupts.

well, without hardware support, at least the SAM ends up
polled on system host PNs. definitely -- unless above management
extension would become part of the next revision.

> As for explaining it, well, the spec is already much more expository
> than normal for such specifications.  We needed to leave some material
> for your thesis!  To say nothing of my consulting business.

i agree. on the other hand, there are already enough "sidenotes" in 
being actually more precise than one might expect.

i partially find this pretty disturbing. after all, the specification
has been built around PICMG2.x, while leaving even the most basic
property -- host vs. peripheral -- more or less undiscussed.
from that standpoint, you guys should then rather have tried to avoid
terms like "CompactPCI" or even "PCI" in general...  :)

> Some hints for the implementation of such an ISR - 
> The SN may know generally which slots have outstanding requests, and
> hence can limit the potential interrupt routes/sources that need to
> execute the "speculative" isr code.
> 
> Using the flush before alert property ensures that the CCB will have a
> state change prior to the arrival of the actual interrupt.  This makes
> the search for potential interrupt sources relatively quick, especially
> if the number of outstanding requests has been limited by the SN.
> 
> The good news is that this situation is transient, and the SN knows
> whether it needs to perform the additional search or not, based upon
> whether the SN has outstanding connection responses.  The problem is
> that there needs to be a sort of general interrupt handler trailer that
> gets conditionally called if the interrupt is not from any known
> source.  Perhaps the isr routine can be added to the line where the
> interrupt will occur as needed, and generally not be part of the isr
> chain when not needed.
> 
> That is - there is a general_214_search_for_new_interrupt_source routine
> that is registerd as an isr for any line on which a bus 214 interrupt
> might occur.  This registration is done as part of the connection
> response prior to entering responder step 4.  Then, an unacknowledge
> interrupt on that line will enter the general search handler.  There
> will be a limited number of potential 214 sources for that interrupt,
> which the general handler will search.  If it finds new management, it
> will set that up and register a new handler, and if there are no more
> pending responses, remove itself from the isr chain.  A bit tricky,
> perhaps, but not a whole lot of overhead, and not intrusive to the rest
> of the interrupt system.  Just watch the SMP stuff.  

(this last sentence i did not understand)

Note that the
> general handler never acks the interrupt, but rather adds the real
> interrupt handler to the list to actually perform the ack after the
> general handler returns.

neat idea. especially since it would allow to shift the whole 214 irq
dispatch chain over to the interrupt system without need to keep
separate (connection-oriented) queues in a driver.

however, for a few subtle reasons, it just won't work :P
i don't know about other operating systems, but the linux interrupt
system ist just not prepared to mess around with the isr chains
from within irq context.
the expectations for conventional device drivers are simple:
- probe for the device
- set it up
- register the interrupt handler
- run for a week, the unregister the handler on driver shutdown.
and do that in process context, e.g. insmod/rmmod.
for 99.9% of all possible hardware this is perfectly o.k. 
for 2.14 obviously not :)

that's what the kernel folks expect to happen, and as a result,
request_irq() is doing GFP_KERNEL memory allocations for it's irqaction
descriptor and therefore will crash when called from a bottom half, let
alone from another isr.
free_irq will hang if you try to release the line from the isr itself.

look into arch/i386/kernel/irq.c. in smp builds, free_irq will block
if called from the isr it's going to unregister (is that what you meant
with "watch smp" above?)
so switching isrs directly from within is simply not possible.
at least not without just rewriting the interrupt system of any
architecture we are going to run on B)

getting process context is not too much of a problem during "normal"
operations. the mcdev interface has been changed in the meanwhile to
allow for a connect() operation, with the very first bus reference (i.e.
RL or SAM) as a parameter. while the connection setup state machines are
still assuming to run from bottom halves, calls to connect() are run
from keventd. this allows the host device to figure out which device the
address refers to and register an isr on the first connection made.
nowadays, i'm thinking whether it would be a better idea to create plain
kernel threads for connection setup, but right now it's just step4 which
gives me a headache.

fwiw, as far as i can tell from here, 2.14 bus interrupt dispatch will
end up with chains maintained by the respective device driver.

i've mostly been asking in order to make sure not to misunderstand
things. obviously i didn't. good to know. thanks so far :)

btw, the 21554 code should be functional (assuming i didn't mess up cvs
contents over the last few days). if you find the time and interest to
try it out let me know. there's still no discovery support and not much
comments in the tree, so i would write a short howto about
"howto-get-it-going" again.
it's now about three month since my last preview release. waaay too much
time. should have done so with the drawbridge drivers. maybe even
before. i thought the hostboard stuff would turn out to be much simpler.
funny :) 

thanks,
dns

-- 
__________________________________________________________________
mailto: st...@in...

Re: [mcnet4l-devel] Re: responder step 4

[Mark H. <mh...@mv...>]

Daniel Stodden wrote:
> 
> hi mark et al.
> 
> On Wed, 2001-12-12 at 16:42, Daniel Stodden wrote:
> 
> > You are not confused at all.  You perfectly understand the difficulties
> > of using an out-of-band interrupt in 2.14.  The committee pondered this
> > at length.  The choice was an awkward ISR implementation, an ad hoc
> > driver for each class of 2.14 hardware that potentially would interfer
> > with other uses of the same hardware node, or a prohibition of
> > out-of-band interrupts.  We chose awkward. Just how awkward depends on
> > the rest of the hardware implementation. If the hardware funnels many
> > interrupts through a particular interrupt port, then locating the
> > initiator node may take some time, although the implementation on the SN
> > can mitigate this by limiting the number of outstanding connection
> > request that it accepts.  And, you are correct, it is possible to lose
> > the use of the interrupt system completely if the PN messes up the
> > management info.  Of course, the PN can normally just scribble all of
> > the SN's memory to much more spectacular effect 8-).  I suppose we could
> > have passed some more information in the initiator CCB - for example, we
> > might say that the local management means are filled out with the
> > management means that would be given if the responder were to ask for a
> > bus interrupt.
> 
> grmpf. fine, i've had a proposal exactly like this in mind while i wrote
> the original email.
> guess i should write more extensive problem descriptions or carsten will
> put _your_ name on my thesis paper.
> 
>   This would probably entail an additional phase to the
> > connection process to excahnge the speculative management means followed
> > by the actual management means.  I will keep that in mind if we get to a
> > second revision of the spec.
> >
> > SNs can avoid this problem by using memory polled alerts or by having
> > hardware which allows inband interrupts.
> 
> well, without hardware support, at least the SAM ends up
> polled on system host PNs. definitely -- unless above management
> extension would become part of the next revision.
Yeah, this is why I got rid of the SNs SAM - if you have an early draft
you'll see that the original RP was for the SN to put its SAM into the
RL, and then have the PNs SAM jam the SN - without interrupt support
this was very ugly.  But if one archtects the system to propagate PNs on
the same hardware node as the SN, then you get back to that problem.

Many hardwares that appear at first glance not to have an in-band
interrupt mechansim actually do - most up systems that have APIC or MPIC
interrupt models have the capability of using a PCI write to one of the
IP registers to generate an interrupt.  I don't now know if that
capability is lost in mp systems or not.
> 
> > As for explaining it, well, the spec is already much more expository
> > than normal for such specifications.  We needed to leave some material
> > for your thesis!  To say nothing of my consulting business.
> 
> i agree. on the other hand, there are already enough "sidenotes" in
> being actually more precise than one might expect.
We were besieged for additional explanations during the ballot process. 
Had it been left to me, I'd have said - oh, you don't understand the
subtle implications, buy my book, or read DNS's thesis 8-)  However, we
got what we got, and only about a year too late.
> 
> i partially find this pretty disturbing. after all, the specification
> has been built around PICMG2.x, while leaving even the most basic
> property -- host vs. peripheral -- more or less undiscussed.
> from that standpoint, you guys should then rather have tried to avoid
> terms like "CompactPCI" or even "PCI" in general...  :)
Ah, but then PICMG probably wouldn't have sponsored the work.
> 
> > Some hints for the implementation of such an ISR -
> > The SN may know generally which slots have outstanding requests, and
> > hence can limit the potential interrupt routes/sources that need to
> > execute the "speculative" isr code.
> >
> > Using the flush before alert property ensures that the CCB will have a
> > state change prior to the arrival of the actual interrupt.  This makes
> > the search for potential interrupt sources relatively quick, especially
> > if the number of outstanding requests has been limited by the SN.
> >
> > The good news is that this situation is transient, and the SN knows
> > whether it needs to perform the additional search or not, based upon
> > whether the SN has outstanding connection responses.  The problem is
> > that there needs to be a sort of general interrupt handler trailer that
> > gets conditionally called if the interrupt is not from any known
> > source.  Perhaps the isr routine can be added to the line where the
> > interrupt will occur as needed, and generally not be part of the isr
> > chain when not needed.
> >
> > That is - there is a general_214_search_for_new_interrupt_source routine
> > that is registerd as an isr for any line on which a bus 214 interrupt
> > might occur.  This registration is done as part of the connection
> > response prior to entering responder step 4.  Then, an unacknowledge
> > interrupt on that line will enter the general search handler.  There
> > will be a limited number of potential 214 sources for that interrupt,
> > which the general handler will search.  If it finds new management, it
> > will set that up and register a new handler, and if there are no more
> > pending responses, remove itself from the isr chain.  A bit tricky,
> > perhaps, but not a whole lot of overhead, and not intrusive to the rest
> > of the interrupt system.  Just watch the SMP stuff.
> 
> (this last sentence i did not understand)

I'll explain below - 
> 
> Note that the
> > general handler never acks the interrupt, but rather adds the real
> > interrupt handler to the list to actually perform the ack after the
> > general handler returns.
> 
> neat idea. especially since it would allow to shift the whole 214 irq
> dispatch chain over to the interrupt system without need to keep
> separate (connection-oriented) queues in a driver.
> 
> however, for a few subtle reasons, it just won't work :P
> i don't know about other operating systems, but the linux interrupt
> system ist just not prepared to mess around with the isr chains
> from within irq context.
> the expectations for conventional device drivers are simple:
> - probe for the device
> - set it up
> - register the interrupt handler
> - run for a week, the unregister the handler on driver shutdown.
> and do that in process context, e.g. insmod/rmmod.
> for 99.9% of all possible hardware this is perfectly o.k.
> for 2.14 obviously not :)
> 
> that's what the kernel folks expect to happen, and as a result,
> request_irq() is doing GFP_KERNEL memory allocations for it's irqaction
> descriptor and therefore will crash when called from a bottom half, let
> alone from another isr.
> free_irq will hang if you try to release the line from the isr itself.

Yeah, I don't think that I would have tried to call the normal
request_irq and free_irq entries to accomplish my devious puposes.  I
haven't actually looked into what is needed, but most likey one would be
required to play directly with the actual list structures that maintain
the interrupt mappings.  This might indeed be archtecture dependent. 
And, if one were to do so, the necessary locking, etc would be needed to
keep the other processors out of the lists while they were being
manipulated.  The source of my SMP remarks.

If you aren't willing to go so far as to seriously hack the interrupt
structures, then an additional level of interrupt handler can be used to
take potential 214 interrupts, and do a sub-dispatch to the ultimate
destination handler.  Then the tables that describe that dispatch are
contained within the single isr, and may be modified by the front end to
the dispatcher as needed.  This, of course, looks like a single 214
interrupt handler (which probably has to be registered on multiple irqs)
to the system.  Then various connection contexts register or request
interrupt connections as the connections occur.

Some of the early designs I played with contemplated that sort of
architecture. It's a lot like any other peripheral with limited
resources managed and assigned as needed by the resource driver.
> 
> look into arch/i386/kernel/irq.c. in smp builds, free_irq will block
> if called from the isr it's going to unregister (is that what you meant
> with "watch smp" above?)
> so switching isrs directly from within is simply not possible.
> at least not without just rewriting the interrupt system of any
> architecture we are going to run on B)
or adding some interfaces!
> 
> getting process context is not too much of a problem during "normal"
> operations. the mcdev interface has been changed in the meanwhile to
> allow for a connect() operation, with the very first bus reference (i.e.
> RL or SAM) as a parameter. while the connection setup state machines are
> still assuming to run from bottom halves, calls to connect() are run
> from keventd. this allows the host device to figure out which device the
> address refers to and register an isr on the first connection made.
> nowadays, i'm thinking whether it would be a better idea to create plain
> kernel threads for connection setup, but right now it's just step4 which
> gives me a headache.
> 
> fwiw, as far as i can tell from here, 2.14 bus interrupt dispatch will
> end up with chains maintained by the respective device driver.

Yes, as mentioned above.  I should probably read to the end, so as not
to influence Carsten too much.  Really Carsten, all I did was throw
stuff from a very high altitude.
> 
> i've mostly been asking in order to make sure not to misunderstand
> things. obviously i didn't. good to know. thanks so far :)
> 
> btw, the 21554 code should be functional (assuming i didn't mess up cvs
> contents over the last few days). if you find the time and interest to
> try it out let me know. there's still no discovery support and not much
> comments in the tree, so i would write a short howto about
> "howto-get-it-going" again.
I wish I could be more optimistic, but my work commitments are not
currently centered on 214, although we will probably move to it if you
get something working well.  In fact, I'm off to hack some IO_APIC code
to accommodate some recent Intel hardware "innovations". 
> it's now about three month since my last preview release. waaay too much
> time. should have done so with the drawbridge drivers. maybe even
> before. i thought the hostboard stuff would turn out to be much simpler.
> funny :) 
Usually the way it turns out.  Keep up the good work.

Mark Huth
> 
> thanks,
> dns
> 
> --
> __________________________________________________________________
> mailto: st...@in...
> 
> _______________________________________________
> mcnet4l-devel mailing list
> mcn...@li...
> https://lists.sourceforge.net/lists/listinfo/mcnet4l-devel