From: Jörn N. <net...@st...> - 2009-10-26 12:06:28
|
hi everyone! bored on a train, part 2: http://subversion.ffado.org/wiki/IrqPriorities still needs some additions, notably the rtirq section is out of date. please comment, and can anyone explain to me why rtc0 should have maximum priority? please also critically review the assumptions i made about the order of tasklets. best, jörn -- Jörn Nettingsmeier Audio engineer Meister für Veranstaltungstechnik (Bühne/Studio) Lortzingstr. 11, D-45128 Essen, Germany +49 177 7937487 http://stackingdwarves.net |
From: Stefan R. <st...@s5...> - 2009-10-31 13:27:32
|
Jörn Nettingsmeier wrote: > hi everyone! > > bored on a train, part 2: > > http://subversion.ffado.org/wiki/IrqPriorities > > still needs some additions, notably the rtirq section is out of date. > please comment, and can anyone explain to me why rtc0 should have > maximum priority? please also critically review the assumptions i made > about the order of tasklets. [Disclaimer: I only superficially know how -rt differs from mainline from what I read at lwn.net and never used -rt myself.] I wonder about nomenclature on the wiki page. In the preamble, "Since most of the actual work that has to be done to respond to an IRQ is done in a tasklet, one should also ensure that the priority of the softirq-tasklet processes is high enough. Otherwise tasklets are not executed in time." sounds correct. However, in section '2. Find out which processes handle the IRQs' and '3. Set the tasklet priorities', there is talk about "IRQ tasklets", and IRQ handler threads and soft-IRQ threads are lumped together. The driver implementation is that ohci1394's IRQ handler thread executes only the minimal ohci_irq_handler() function which reads and writes the controller's IRQ event registers and schedules e.g. IR and IT tasklets for execution. Tasklets on the other hand are work items which are performed in the context of a soft-IRQ thread --- I guess that's [sirq-tasklet/#], where # is presumably the CPU which also executes the OHCI 1394 controller's IRQ handler. So, there are "hardware interrupts" (i.e. the IRQ handlers) and "software interrupts" (timer interrupts; in -rt apparently split into a few more or less special-purpose threads. From the listings at your wiki page it appears that -rt creates one soft-IRQ thread per CPU (per core) as a catch-all tasklet execution thread ("tasklet" = a work item to be done at the next opportunity). So, I think a terminology like "IRQ handlers" on one hand and "tasklets" or "soft IRQs" on the other hand should be used. Regarding how to set IRQ priorities: The current implementation obviously has the drawback that ohci1394 uses the tasklet API together with a number of other kernel subsystems. (Ditto the firewire-ohci driver.) Thus you have no means to e.g. prioritize FireWire IR and IT jobs over whatever other tasklets the system might run (and vice versa, possibly prioritize whatever fundamental other system tasklets over FireWire IR and IT jobs). The way out is probably to switch firewire-ohci to the threaded IRQ handler API which is now already available in mainline Linux and move the tasklets into the IRQ handler. (This is to be done after somebody fixed basic IR/IT of the firewire-ohci + firewire-core + libraw1394 + ffado stack of course... unless the firewire-ohci + ~-core + ALSA streaming interface driver + ffado work is finished even before that...) Besides the currently close-to-zero speed of FFADO-targeted kernel driver development (sorry for that; I somehow need to get back some free time for driver hacking), there might be more challenges: - How will mainline kernel's (and mainstream distro kernels') FireWire I/O performance turn out if the firewire stack starts using an IRQ handler thread while some other subsystems still don't for some time? - It may be desirable or necessary to reorganize the subystem's controller event handling further --- move some of what is currently a tasklet into a different context than the IRQ handler thread; vice versa move some work which is currently done outside tasklets into the IRQ handler thread... -- Stefan Richter -=====-==--= =-=- ===== http://arcgraph.de/sr/ |
From: Jörn N. <net...@st...> - 2009-11-01 14:35:09
|
hi stefan! thanks for your feedback! please bear with a number of very stupid questions: Stefan Richter wrote: > [Disclaimer: I only superficially know how -rt differs from mainline > from what I read at lwn.net and never used -rt myself.] in -rt, every IRQ handler is threaded by default and running at FF/50. > I wonder about nomenclature on the wiki page. In the preamble, "Since > most of the actual work that has to be done to respond to an IRQ is done > in a tasklet, one should also ensure that the priority of the > softirq-tasklet processes is high enough. Otherwise tasklets are not > executed in time." sounds correct. that snippet was in the previous version already. > However, in section '2. Find out > which processes handle the IRQs' and '3. Set the tasklet priorities', > there is talk about "IRQ tasklets", and IRQ handler threads and soft-IRQ > threads are lumped together. is there a difference? can you explain? from my understanding, "tasklets" are what used to be "bottom halves", i.e. the part of an irq handler that actually deals with the work at hand, while the "top half" only acknowledges the IRQ and then passes the work on to the bottom half... but then, that's what i caught in passing, i'm not really in the know. > The driver implementation is that ohci1394's IRQ handler thread executes > only the minimal ohci_irq_handler() function which reads and writes the > controller's IRQ event registers and schedules e.g. IR and IT tasklets > for execution. what is IR and IT? > Tasklets on the other hand are work items which are performed in the > context of a soft-IRQ thread --- I guess that's [sirq-tasklet/#], where > # is presumably the CPU which also executes the OHCI 1394 controller's > IRQ handler. ps -eLo pid,cls,rtprio,nice,command | grep "irq" 3 FF 49 - [sirq-high/0] 4 FF 49 - [sirq-timer/0] 5 FF 49 - [sirq-net-tx/0] 6 FF 49 - [sirq-net-rx/0] 7 FF 49 - [sirq-block/0] 8 FF 49 - [sirq-tasklet/0] 9 FF 49 - [sirq-sched/0] 10 FF 49 - [sirq-hrtimer/0] 11 FF 49 - [sirq-rcu/0] 22 FF 50 - [irq/9-acpi] 32 FF 50 - [irq/1-i8042] 74 FF 50 - [irq/23-sata_nv] 78 FF 50 - [irq/22-sata_nv] 85 FF 50 - [irq/14-pata_amd] 86 FF 50 - [irq/15-pata_amd] 293 FF 50 - [irq/21-ehci_hcd] 404 FF 50 - [irq/20-ohci_hcd] 531 FF 50 - [irq/23-ohci_hcd] 1125 FF 50 - [irq/8-rtc0] 1271 FF 50 - [irq/19-ohci1394] 1279 FF 50 - [irq/17-rme9652] 1308 FF 50 - [irq/18-EMU10K1] 1325 FF 50 - [irq/21-NVidia C] 2390 FF 50 - [irq/22-eth0] 2651 FF 50 - [irq/16-eth1] 15779 TS - 0 grep irq > So, there are "hardware interrupts" (i.e. the IRQ handlers) and > "software interrupts" (timer interrupts; in -rt apparently split into a > few more or less special-purpose threads. From the listings at your > wiki page it appears that -rt creates one soft-IRQ thread per CPU (per > core) as a catch-all tasklet execution thread ("tasklet" = a work item > to be done at the next opportunity). > > So, I think a terminology like "IRQ handlers" on one hand and "tasklets" > or "soft IRQs" on the other hand should be used. so you say that those things running in sirq-tasklet should be called a tasklet? and all the [irq/foo] kernel threads are "IRQ handlers"? > Regarding how to set IRQ priorities: > > The current implementation obviously has the drawback that ohci1394 uses > the tasklet API together with a number of other kernel subsystems. > (Ditto the firewire-ohci driver.) Thus you have no means to e.g. > prioritize FireWire IR and IT jobs over whatever other tasklets the > system might run (and vice versa, possibly prioritize whatever > fundamental other system tasklets over FireWire IR and IT jobs). > > The way out is probably to switch firewire-ohci to the threaded IRQ > handler API which is now already available in mainline Linux and move > the tasklets into the IRQ handler. (This is to be done after somebody > fixed basic IR/IT of the firewire-ohci + firewire-core + libraw1394 + > ffado stack of course... unless the firewire-ohci + ~-core + ALSA > streaming interface driver + ffado work is finished even before that...) aren't all IRQs threaded by default on rt kernels? best, jörn -- Jörn Nettingsmeier Meister für Veranstaltungstechnik Audio and event engineer Ambisonic surround recordings http://stackingdwarves.net +49 177 7937487 |
From: Stefan R. <st...@s5...> - 2009-11-01 17:20:39
|
Jörn Nettingsmeier wrote: > Stefan Richter wrote: [...] >> However, in section '2. Find out >> which processes handle the IRQs' and '3. Set the tasklet priorities', >> there is talk about "IRQ tasklets", and IRQ handler threads and soft-IRQ >> threads are lumped together. > > is there a difference? can you explain? from my understanding, > "tasklets" are what used to be "bottom halves", i.e. the part of an irq > handler that actually deals with the work at hand, while the "top half" > only acknowledges the IRQ and then passes the work on to the bottom > half... but then, that's what i caught in passing, i'm not really in the > know. That's correct. Though more precisely, "tasklet" is one of the kernel's APIs for delayed execution. It happens to be the primary API used to implement bottom halves. (And vice versa, the primary use of this API is to implement bottom halves of interrupt handlers.) In case of the ieee1394 subsystem and firewire subsystem, interrupt event handling typically happens in three rather than two stages: The hardware interrupt service routine schedules a tasklet (for most types of interrupt event type, e.g. isochronous reception, asynchronous transmission...). In several cases, the tasklet then wakes up a kernel thread or a userspace process to do some more work. Of course it is subjective whether this third stage is still seen as part of interrupt handling; though functionally it is. The ieee1394 subsystem uses dedicated kernel threads for completion of outbound asynchronous transactions and for bus reset handling (bus management, device discovery). The firewire subsystem uses shared kernel threads for bus reset handling. So, those kernel threads are not interesting in terms of FireWire audio stream I/O. Audio I/O interrupt handling's third stage is FFADO/ Jack userspace. But back to your question: http://www.wil.cx/matthew/lca2003/paper.pdf gives a lot of background to how previous kernels did and AFAIU current mainline kernel still does operate. Fundamentally, there are three execution contexts: Interrupt, bottom half, process context. Tasklets and soft IRQs are normally executed in bottom half context; in case of high load they are shifted into a dedicated high priority process context. In the -rt kernel, almost everything but the very basic top halves was moved straight into process contexts. Still AFAIU. >> The driver implementation is that ohci1394's IRQ handler thread executes >> only the minimal ohci_irq_handler() function which reads and writes the >> controller's IRQ event registers and schedules e.g. IR and IT tasklets >> for execution. > > what is IR and IT? Isochronous reception, isochronous transmission. >> Tasklets on the other hand are work items which are performed in the >> context of a soft-IRQ thread --- I guess that's [sirq-tasklet/#], where >> # is presumably the CPU which also executes the OHCI 1394 controller's >> IRQ handler. [I guessed this from the name of this -rt kernel thread. :-) ] [...] >> So, there are "hardware interrupts" (i.e. the IRQ handlers) and >> "software interrupts" (timer interrupts; in -rt apparently split into a [Correction: Soft IRQs and timer interrupts are different beasts.] >> few more or less special-purpose threads. From the listings at your >> wiki page it appears that -rt creates one soft-IRQ thread per CPU (per >> core) as a catch-all tasklet execution thread ("tasklet" = a work item >> to be done at the next opportunity). >> >> So, I think a terminology like "IRQ handlers" on one hand and "tasklets" >> or "soft IRQs" on the other hand should be used. > > so you say that those things running in sirq-tasklet should be called a > tasklet? and all the [irq/foo] kernel threads are "IRQ handlers"? That's what I said yesterday; today I'd say it a bit differently. :-) "Interrupt handling" in the most abstract sense covers everything from hardware interrupt service routine (top half) through bottom half to corresponding processes. With "IRQ handler" a few lines above, I really only meant the hardware interrupt service routine (top half). "Tasklet" is a special term which refers to the most common (but not the only) form of implementation of bottom halves. OTOH, a tasklet can never implement a top half. I guess from the little which I read about -rt and from the names of the kernel threads which you listed, that the threads named [irq/...] took over the execution of the mainline kernel's drivers part of the top half, and the [sirq-...] took over the mainline's kernels bottom half execution contexts (soft IRQ and tasklet execution contexts). The tasklet kernel API apparently makes use of [sirq-tasklet/...] kernel threads. >> Regarding how to set IRQ priorities: >> >> The current implementation obviously has the drawback that ohci1394 uses >> the tasklet API together with a number of other kernel subsystems. >> (Ditto the firewire-ohci driver.) Thus you have no means to e.g. >> prioritize FireWire IR and IT jobs over whatever other tasklets the >> system might run (and vice versa, possibly prioritize whatever >> fundamental other system tasklets over FireWire IR and IT jobs). >> >> The way out is probably to switch firewire-ohci to the threaded IRQ >> handler API which is now already available in mainline Linux and move >> the tasklets into the IRQ handler. (This is to be done after somebody >> fixed basic IR/IT of the firewire-ohci + firewire-core + libraw1394 + >> ffado stack of course... unless the firewire-ohci + ~-core + ALSA >> streaming interface driver + ffado work is finished even before that...) > > aren't all IRQs threaded by default on rt kernels? The drawback which I describe here, coming from the ohci1394 driver implementation, affects -rt this way: ohci1394 as well as many other drivers use the tasklet API to perform bottom halves. All of these tasklets are apparently injected into the [sirq-tasklet/...] thread of which there seem to be only as many as there are CPUs (CPU cores). Since ohci1394 isochronous receive tasklet and isochronous transmit tasklet sit between ohci1394's top half and FFADO/ Jack userspace, I conclude that latency of tasklet execution is important for audio latency. However, while -rt lets you control scheduling of ohci1394's top half and of FFADO/ Jack userspace specifically by means of scheduling class and scheduling priority of the corresponding dedicated kernel threads and userspace processes, you only have one global knob for the scheduling priority of any and all tasklets on the system. (Furthermore, ohci1394 I/O tasklets cannot preempt other tasklets on the same CPU core that might be less important to you.) Two possible solutions: - Get ohci1394 its own dedicated tasklet thread in -rt. I think this is unrealistic; the -rt maintainers would probably tell us to do the following instead. - Change ohci1394 (actually firewire-ohci, since we won't fundamentally change ohci1394 anymore) to _not_ use the tasklet API, at least for isochronous I/O. Instead use a process context (a kernel thread, or userspace where it makes sense). This could be one of the traditional kernel thread related APIs, notably the workqueue API, or the threaded interrupt handler API which is present in the mainline since 2.6.30: http://lwn.net/Articles/302043/ http://lxr.linux.no/#linux+v2.6.30/kernel/irq/manage.c#L831 -- Stefan Richter -=====-==--= =-== ----= http://arcgraph.de/sr/ |
From: Jörn N. <net...@st...> - 2009-11-02 20:00:56
|
hi stefan! thanks for your valuable contribution. i hope i've understood it correctly. Stefan Richter wrote: > Jörn Nettingsmeier wrote: >> Stefan Richter wrote: > [...] >>> However, in section '2. Find out >>> which processes handle the IRQs' and '3. Set the tasklet priorities', >>> there is talk about "IRQ tasklets", and IRQ handler threads and soft-IRQ >>> threads are lumped together. >> is there a difference? can you explain? from my understanding, >> "tasklets" are what used to be "bottom halves", i.e. the part of an irq >> handler that actually deals with the work at hand, while the "top half" >> only acknowledges the IRQ and then passes the work on to the bottom >> half... but then, that's what i caught in passing, i'm not really in the >> know. > > That's correct. Though more precisely, "tasklet" is one of the kernel's > APIs for delayed execution. It happens to be the primary API used to > implement bottom halves. (And vice versa, the primary use of this API > is to implement bottom halves of interrupt handlers.) i've tried to work this into http://subversion.ffado.org/wiki/IrqPriorities. i'd appreciate if you and other ffado-devel patrons could verify it for correctness eventually. > Since ohci1394 isochronous receive tasklet and isochronous transmit > tasklet sit between ohci1394's top half and FFADO/ Jack userspace, I > conclude that latency of tasklet execution is important for audio latency. ok. i've noted that, but was reluctant to actually raise its priority. can anybody fill in this blank and add a good recommendation? maybe it should be somewhere between the audio-specific IRQs and the normal IRQs? then again, the default is 49, which is below any IRQs, so i don't know if raising it can mess things up... > However, while -rt lets you control scheduling of ohci1394's top half > and of FFADO/ Jack userspace specifically by means of scheduling class > and scheduling priority of the corresponding dedicated kernel threads > and userspace processes, you only have one global knob for the > scheduling priority of any and all tasklets on the system. (Furthermore, > ohci1394 I/O tasklets cannot preempt other tasklets on the same CPU core > that might be less important to you.) > > Two possible solutions: > - Get ohci1394 its own dedicated tasklet thread in -rt. I think this > is unrealistic; the -rt maintainers would probably tell us to do the > following instead. > - Change ohci1394 (actually firewire-ohci, since we won't > fundamentally change ohci1394 anymore) to _not_ use the tasklet API, > at least for isochronous I/O. Instead use a process context (a > kernel thread, or userspace where it makes sense). This could be > one of the traditional kernel thread related APIs, notably the > workqueue API, or the threaded interrupt handler API which is > present in the mainline since 2.6.30: > http://lwn.net/Articles/302043/ > http://lxr.linux.no/#linux+v2.6.30/kernel/irq/manage.c#L831 sounds interesting. however, i'm quite comfortable in the shallow end of the programmer's pool and would like to remain here for another year or so. and over yonder, it looks like there's fins sticking out of the water... -- Jörn Nettingsmeier Meister für Veranstaltungstechnik Audio and event engineer Ambisonic surround recordings http://stackingdwarves.net +49 177 7937487 |
From: Stefan R. <st...@s5...> - 2009-11-02 21:38:50
|
Jörn Nettingsmeier wrote: > Stefan Richter wrote: [...] > i've tried to work this into > http://subversion.ffado.org/wiki/IrqPriorities. i'd appreciate if you > and other ffado-devel patrons could verify it for correctness eventually. Looks good to me. >> Since ohci1394 isochronous receive tasklet and isochronous transmit >> tasklet sit between ohci1394's top half and FFADO/ Jack userspace, I >> conclude that latency of tasklet execution is important for audio latency. > > ok. i've noted that, but was reluctant to actually raise its priority. > can anybody fill in this blank and add a good recommendation? > > maybe it should be somewhere between the audio-specific IRQs and the > normal IRQs? > then again, the default is 49, which is below any IRQs, so i don't know > if raising it can mess things up... I suppose the tasklet thread should not get higher priority than any (top half) IRQ thread, probably also not higher than some of the other soft IRQ threads. It may or may not be beneficial to prioritize the block layer's soft IRQ and networking soft IRQs lower than the other soft IRQs since they deal with asynchronous things that don't require very low latency. However, unless somebody finds actual evidence that such changes would improve things, these priorities are better left untouched. -- Stefan Richter -=====-==--= =-== ---=- http://arcgraph.de/sr/ |