From: Andrew M. <ak...@zi...> - 2001-11-17 03:18:02
|
If a CPU take an interrupt while holding a contended spinlock, the cost of the interrupt is borne not only by the CPU which takes it, but also by all the CPUs which are waiting for the lock. So here's an x86 patch which disables cpu-local interrupts whenever that CPU is holding a spinlock. I don't have enough CPUs to measure its effects, but perhaps someone who has an 8-way may be interested in trying it with a lock- and interrupt-intensive workload, see what happens? --- linux-2.4.15-pre5/include/asm-i386/spinlock.h Mon Nov 5 21:01:12 2001 +++ linux-akpm/include/asm-i386/spinlock.h Fri Nov 16 18:58:49 2001 @@ -25,6 +25,7 @@ extern int printk(const char * fmt, ...) typedef struct { volatile unsigned int lock; + unsigned long flags; #if SPINLOCK_DEBUG unsigned magic; #endif @@ -79,15 +80,18 @@ typedef struct { static inline void spin_unlock(spinlock_t *lock) { + unsigned long flags; #if SPINLOCK_DEBUG if (lock->magic != SPINLOCK_MAGIC) BUG(); if (!spin_is_locked(lock)) BUG(); #endif + flags = lock->flags; __asm__ __volatile__( spin_unlock_string ); + local_irq_restore(flags); } #else @@ -99,6 +103,7 @@ static inline void spin_unlock(spinlock_ static inline void spin_unlock(spinlock_t *lock) { + unsigned long flags; char oldval = 1; #if SPINLOCK_DEBUG if (lock->magic != SPINLOCK_MAGIC) @@ -106,9 +111,11 @@ static inline void spin_unlock(spinlock_ if (!spin_is_locked(lock)) BUG(); #endif + flags = lock->flags; __asm__ __volatile__( spin_unlock_string ); + local_irq_restore(flags); } #endif @@ -116,11 +123,15 @@ static inline void spin_unlock(spinlock_ static inline int spin_trylock(spinlock_t *lock) { char oldval; + int ret; __asm__ __volatile__( "xchgb %b0,%1" :"=q" (oldval), "=m" (lock->lock) :"0" (0) : "memory"); - return oldval > 0; + ret = (oldval > 0); + if (ret) + local_irq_save(lock->flags); + return ret; } static inline void spin_lock(spinlock_t *lock) @@ -136,6 +147,7 @@ printk("eip: %p\n", &&here); __asm__ __volatile__( spin_lock_string :"=m" (lock->lock) : : "memory"); + local_irq_save(lock->flags); } |
From: george a. <ge...@mv...> - 2001-11-17 08:59:43
|
Andrew Morton wrote: > > If a CPU take an interrupt while holding a contended spinlock, > the cost of the interrupt is borne not only by the CPU which takes > it, but also by all the CPUs which are waiting for the lock. > > So here's an x86 patch which disables cpu-local interrupts > whenever that CPU is holding a spinlock. > > I don't have enough CPUs to measure its effects, but perhaps > someone who has an 8-way may be interested in trying it with > a lock- and interrupt-intensive workload, see what happens? > Too bad you can't set it up so that the interrupts are turned off on the holders machine ONLY when the lock is contested. (A trick that, at the very least, requires knowing which cpu holds the lock, not to mention cross cpu control of the interrupt system.) Maybe a better way would be to cause the APIC to direct the interrupt to a cpu that is not holding a lock (like possibly the cpu that is spinning). This may be something that could be done in the spin loop. Any other ideas, lets see, how about taking the interrupt, noticing that a spin lock is held and passing the interrupt to another cpu, or, possibly, queuing the interrupt until the lock is released. But, we know some spin locks are held for several ms.... Of all these, the one, IMHO, has the most promise is the spin modification to redirect interrupts to the spinning cpu. The work is all done by a cpu that is doing nothing else and the impact on the holder of the lock is nil. I think performance will be hit rather hard by this as the interrupt off/on instructions are pipe line flushers. Doing this on every spin lock will, I think, cause a real hit. -- George ge...@mv... High-res-timers: http://sourceforge.net/projects/high-res-timers/ Real time sched: http://sourceforge.net/projects/rtsched/ |
From: Anton B. <an...@sa...> - 2001-11-18 23:43:01
|
Hi George, > Maybe a better way would be > to cause the APIC to direct the interrupt to a cpu that is not holding a > lock (like possibly the cpu that is spinning). Isnt this what Andrews patch ends up doing? (all cpus holding locks will have irqs disabled) Actually you would also have to modify the spin_lock_irq* to spin with interrupts enabled. > I think performance will be hit rather hard by this as the interrupt > off/on instructions are pipe line flushers. Doing this on every spin > lock will, I think, cause a real hit. If this is the case you might get a performance increase from doing soft disable (outlined in my last email) since there are very many uses of spin_lock_irq*/spin_unlock_irq*. Anton |
From: Gerrit H. <ge...@us...> - 2001-11-19 01:45:38
|
> Hi George, > > > Maybe a better way would be > > to cause the APIC to direct the interrupt to a cpu that is not holding a > > lock (like possibly the cpu that is spinning). > > Isnt this what Andrews patch ends up doing? (all cpus holding locks will > have irqs disabled) Actually you would also have to modify the > spin_lock_irq* to spin with interrupts enabled. On Intel, there are two levels of ignoring interrupts, in part because there is an external interrupt controller, the IO APIC which is not related to the CPUs ability to ignore interrupts. Most interrupts are disabled at the CPU only (using cli/sti instructions) but the IO APIC isn't typically informed of the CPUs choice to ignore interrupts. So, the IO APIC may still select to route an interrput to a CPU which will ignore that interrupt. Dave Olien's mechanism (based on DYNIX/ptx) tells the IO APIC that the CPU is unwilling (or less willing) to accept an interrupt. Then the IO APIC modifies its delivery of the interrupt to the CPU most willing to accept an interrupt (round robin-ing between those at the same temporary priority level). > > I think performance will be hit rather hard by this as the interrupt > > off/on instructions are pipe line flushers. Doing this on every spin > > lock will, I think, cause a real hit. > > If this is the case you might get a performance increase from doing soft > disable (outlined in my last email) since there are very many uses of > spin_lock_irq*/spin_unlock_irq*. > > Anton I don't think cli/sti are as heavyweight of a sync point on IA32, so different mechanisms for disabling/ignoring/re-routing interrupts are likely to have enormously different behaviors between IA32 and PPC. gerrit |
From: Anton B. <an...@sa...> - 2001-11-19 02:05:47
|
> On Intel, there are two levels of ignoring interrupts, in part > because there is an external interrupt controller, the IO APIC > which is not related to the CPUs ability to ignore interrupts. Most > interrupts are disabled at the CPU only (using cli/sti instructions) > but the IO APIC isn't typically informed of the CPUs choice to ignore > interrupts. So, the IO APIC may still select to route an interrput to > a CPU which will ignore that interrupt. So I guess there is no way to tell the APIC to redirect that interrupt? > Dave Olien's mechanism (based > on DYNIX/ptx) tells the IO APIC that the CPU is unwilling (or less > willing) to accept an interrupt. Then the IO APIC modifies its delivery > of the interrupt to the CPU most willing to accept an interrupt (round > robin-ing between those at the same temporary priority level). Does it take many cycles to read from/write to the external APIC? For quick spinlocks it could hurt to have to update the APIC. Another thing I have been wanting to try is to distribute interrupts to cpus in the idle thread if possible (we can increase the cpus priority in the interrupt controller). > I don't think cli/sti are as heavyweight of a sync point on IA32, so > different mechanisms for disabling/ignoring/re-routing interrupts are > likely to have enormously different behaviors between IA32 and PPC. Yep, I mentioned it in part because one of the guys here thinks the intel guys were talking about a similar scheme a while ago on linux-kernel. Anton |
From: Gerrit H. <ge...@us...> - 2001-11-19 03:33:17
|
> > On Intel, there are two levels of ignoring interrupts, in part > > because there is an external interrupt controller, the IO APIC > > which is not related to the CPUs ability to ignore interrupts. Most > > interrupts are disabled at the CPU only (using cli/sti instructions) > > but the IO APIC isn't typically informed of the CPUs choice to ignore > > interrupts. So, the IO APIC may still select to route an interrput to > > a CPU which will ignore that interrupt. > > So I guess there is no way to tell the APIC to redirect that interrupt? Nope - just to say "I want it less" or "I want it more". The IO APIC uses the priorities to make the final decision. > > Dave Olien's mechanism (based > > on DYNIX/ptx) tells the IO APIC that the CPU is unwilling (or less > > willing) to accept an interrupt. Then the IO APIC modifies its delivery > > of the interrupt to the CPU most willing to accept an interrupt (round > > robin-ing between those at the same temporary priority level). > > Does it take many cycles to read from/write to the external APIC? For > quick spinlocks it could hurt to have to update the APIC. This isn't a rewrite of the APIC mappings, but simply an update of the TPR register. I don't know what the cycle count is but I don't think it is as expensive as a write to IO space. I'll try to get a more accurate cycle count when I'm in the office tomorrow. > Another thing I have been wanting to try is to distribute interrupts to > cpus in the idle thread if possible (we can increase the cpus priority in > the interrupt controller). That is what Dave's patch does for Intel through the use of priorities. Idle CPUs maximize their chances of getting an interrupt, those in user level code are second in priority class. Those in kernel mode are third, those handling interrupts (with interrupts unblocked) are fourth, and those blocking interrupts are last. At least, from memory, I think that's what we last decided. If not, Dave will correct me tomorrow. ;) > > I don't think cli/sti are as heavyweight of a sync point on IA32, so > > different mechanisms for disabling/ignoring/re-routing interrupts are > > likely to have enormously different behaviors between IA32 and PPC. > > Yep, I mentioned it in part because one of the guys here thinks the > intel guys were talking about a similar scheme a while ago on linux-kernel. > > Anton gerrit |
From: Dipankar S. <dip...@in...> - 2001-11-20 08:49:52
|
On Sun, Nov 18, 2001 at 07:32:41PM -0800, Gerrit Huizenga wrote: > > > Dave Olien's mechanism (based > > > on DYNIX/ptx) tells the IO APIC that the CPU is unwilling (or less > > > willing) to accept an interrupt. Then the IO APIC modifies its delivery > > > of the interrupt to the CPU most willing to accept an interrupt (round > > > robin-ing between those at the same temporary priority level). > > > > Does it take many cycles to read from/write to the external APIC? For > > quick spinlocks it could hurt to have to update the APIC. > > This isn't a rewrite of the APIC mappings, but simply an update > of the TPR register. I don't know what the cycle count is but I don't > think it is as expensive as a write to IO space. I'll try to get > a more accurate cycle count when I'm in the office tomorrow. Not sure what the cost is likely to be on ia32, but on ia64 it is likely to be higher. To redirect interrrupts to other CPUs, you will probably need to update the XTPR too which is in the chipsets. I have been told that the cost of updating TPR + XTPR is 70-100 cycles. Thanks Dipankar -- Dipankar Sarma <dip...@in...> http://lse.sourceforge.net Linux Technology Center, IBM Software Lab, Bangalore, India. |
From: george a. <ge...@mv...> - 2001-11-19 18:20:33
|
Gerrit Huizenga wrote: > ~snip > > I don't think cli/sti are as heavyweight of a sync point on IA32, so > different mechanisms for disabling/ignoring/re-routing interrupts are > likely to have enormously different behaviors between IA32 and PPC. > Gosh I would have though differently, but, lets see, does IA32 do OOE. Last I heard it did not, so that would allow faster cli/sti, still it is a sync point, if not a pipe line flush. -- George ge...@mv... High-res-timers: http://sourceforge.net/projects/high-res-timers/ Real time sched: http://sourceforge.net/projects/rtsched/ |
From: Anton B. <an...@sa...> - 2001-11-18 23:37:23
|
Hi Andrew, > If a CPU take an interrupt while holding a contended spinlock, > the cost of the interrupt is borne not only by the CPU which takes > it, but also by all the CPUs which are waiting for the lock. > > So here's an x86 patch which disables cpu-local interrupts > whenever that CPU is holding a spinlock. > > I don't have enough CPUs to measure its effects, but perhaps > someone who has an 8-way may be interested in trying it with > a lock- and interrupt-intensive workload, see what happens? Interesting, I will be sure to test it on the 16 way ppc64 when I get back to Canberra later in the week. I know at least on ppc64, enabling or disabling interrupts is quite expensive (it has to be context synchronising on most cpus which requires the cpu to stall and the pipeline to be flushed I think). Paul Mackerras has been working on soft disable for ppc64 which means __cli and __sti just set and clear a bit and if an interrupt comes in we can tell the interrupt hardware to deliver it to another cpu instead. Does x86 have the ability to do the same? Lastly at the moment we disable interrupts then spin waiting to take the spinlock. I wonder if there will be an advantage to reenabling interrupts while spinning. This would be quite lightweight when combined with soft disable. Anton |