[Lse-tech] local_irq_disable() and highly-contended locks

[John H. <ha...@sg...>]

Folks,

One thing we're seeing here on systems with lots of CPUs (e.g., 32 or
64) and workloads that produce significant contention on some key locks
(e.g., tasklist_lock in 2.4.19, or one of the runqueue spinlocks, or ...
pick one) is the unfortunate effect of a spin_lock_irq() or
write_lock_irq(): disable interrupts, then contend for the lock.  If the
lock is highly contended and we encounter high wait-times, then
interrupts are disabled for the entire wait-time, which can be a
distressingly significant period of time.

I have observed some interrupts-disabled periods of several hundred
milliseconds for some locks, and many occurrences of 10-30 milliseconds
of interrupts-disabled periods for several locks.  (SGI Itanium 2 NUMA
systems, to be specific.)

An effective solution is an interrupt-friendly locking scheme, such as:
  do { disable_interrupts;
         if (try_to_acquire_lock)
             break;
         else
              enable_interrupts;
  } while (1)

Why isn't this a generic replacement for the current *_lock_irq*
routines?  Is it simply the extra overhead (and slightly larger size) of
the looping code?  I'm not seeing any significant performance
degradation using this friendlier locking scheme.

(I should also note that I am employing this scheme concurrent with a
write_trylock() and a spin_trylock() that gently sniffs at the lock to
determine its current state, rather than starting off with the relative
heavyweight cmpxchg.)

Comments, please!

John Hawkes
ha...@sg...