From: John Hawkes <hawkes@en...> - 2001-03-09 02:37:29
I ran "reflex" against the baseline mips64 2.4.1 kernel + lockmeter to
examine contention on the runqueue_lock.
For the 32p configuration and the particular "reflex" workload
configuration I used, there is 98% contention on the runqueue_lock.
That is, essentially every time schedule() on a CPU attempts to acquire
the runqueue_lock, that lock is already owned.
The mean hold-time is 57 usecs (predominantly from places in
schedule()), with a max hold-time of 261 usec. The mean wait-time is
almost 2.4 *milliseconds*, with a max wait-time of 276 *milliseconds* --
more than a quarter-second.
This is typical of hot locks in a high-cpu-count NUMA system: once the
contention rate exceeds some nominal value, then the wait-time
(especially the max wait-time) deteriorates nonlinearly. The nodes
furthest away from the physical memory -- those with the longest access
latencies -- get starved out when the herd of waiters are circling the
spinlock, waiting for the owner to release.