Steps to reproduce:
(1) use squashfs as the root fs
(2) run a few apps with high oom_score on the backgroun (e.g. boinc)
(3) run an app which allocate too much memory
oom killer will try to kill apps from (2), but the kernel stuck. If I compile the kernel with some debug info, several lock-related messages from the squashfs module are displayed before the kernel stuck. When I replaced the root fs with ext2, the bug is not appeared.
tested kernels: 2.6.26, 2.6.30, 2.6.32 (from Debian distribution)
To reproduce this bug, you can use the following script to simulate app with the high oom score (you have to run several of them!):
#!/bin/sh
echo 15 > /proc/$$/oom_adj
while true; do true; done
and the following code to allocate the memory:
main() { while(1) if (!malloc(1024)) exit(1); }
I'm the reporter of the bug (I forgot too login before I fill the bug report). Fell free to ask me if you need an additional informations.
Fixed in kernel 3.3, upstream commit hash e552a596687bf0e1802c744a7bb113afbd2bf4d4