|
From: Peter W. <pe...@we...> - 2008-11-12 00:44:43
|
I've been working on porting valgrind (trunk) to FreeBSD on and off
for a while. I've got it running fairly happily on x86 and amd64
platforms. The integration of AIX support set me back a bit but I've
finally caught up to trunk again.
Anyway, the big problem I'm running into right now is interactions
between VEX, and FreeBSD's threading library. Implementation-wise it
is closer to linux nptl than linuxthreads, but the problem is the
locking primitives.
FreeBSD uses something (umtx) that appears similar to futex at first
glance, but when it comes to the crunch, a couple of things about umtx
have me quite stumped.
Take the following tiny bit of code. The lock is attempted in
userland, then falls back to kernel assistance:
thr_umtx.h:
static inline int
_thr_umtx_lock(volatile umtx_t *mtx, long id)
{
if (atomic_cmpset_acq_ptr((volatile uintptr_t *)mtx,
(uintptr_t)UMTX_UNOWNED, (uintptr_t)id))
return (0);
return (__thr_umtx_lock(mtx, id));
}
thr_umtx.c:
int
__thr_umtx_lock(volatile umtx_t *mtx, long id)
{
while (_umtx_op((struct umtx *)mtx, UMTX_OP_LOCK, id, 0, 0))
;
return (0);
}
This is over simplified, but if there is contention, it is handled by
the kernel (_umtx_op() is the syscall).
The problem I'm having is that umtx does some more complex operations,
including posix-ish condvar support. ie: the syscall does atomic
operations on both a umtx and a ucond at the same time. This seems to
be causing bad problems with the lack of libvex's atomic/locked
instruction support.
The scenario I think I'm running into goes something like this:
Thread 1:
begin with BigLock held
attempts atomic op, fails.
performs umtx_op syscall, which goes async.
async handler releases BigLock, calls kernel.
Meanwhile, Thread 2, on another cpu:
acquires BigLock
does atomic ops via VEX.
At this point, both VEX and the kernel are actively reading/writing
the same umtx on different cpu cores. VEX isn't using locked or
atomic cycles and starts losing races with the kernel which is. Bad
things happen.
I am not 100% sure that I haven't messed something else up as well,
but this has been bothering me for a while. Does my explanation make
sense? Am I understanding things correctly?
--
Peter Wemm - pe...@we...; peter@FreeBSD.org; pe...@ya...; KI6FJV
"All of this is for nothing if we don't go to the stars" - JMS/B5
"If Java had true garbage collection, most programs would delete
themselves upon execution." -- Robert Sewell
|
|
From: Julian S. <js...@ac...> - 2008-11-12 13:18:26
|
> At this point, both VEX and the kernel are actively reading/writing > the same umtx on different cpu cores. VEX isn't using locked or > atomic cycles and starts losing races with the kernel which is. Bad > things happen. > > I am not 100% sure that I haven't messed something else up as well, > but this has been bothering me for a while. Does my explanation make > sense? Am I understanding things correctly? Yes, I think this is a correct analysis. It's an interesting question why it works (or appears to work :-) on Linux; perhaps the kernel's helper syscall (futex) doesn't do stuff atomically. I can't see any way to fix this apart from to implement atomics properly in Vex. That's a big job because it involves messing with the IR, the IR optimiser, and all the tools. It'd also need to take into account how to support the ppc lwarx/stwcx stuff properly. However, it's something I've been considering doing anyway, since the lack of atomics support will bite increasing in our increasingly parallel world. J |
|
From: Paul M. <pa...@sa...> - 2008-11-12 22:10:27
|
Julian Seward writes: > Yes, I think this is a correct analysis. It's an interesting question > why it works (or appears to work :-) on Linux; perhaps the kernel's > helper syscall (futex) doesn't do stuff atomically. The original futex call was simply "sleep until the value at location x is not equal to y". The reason for not having the kernel do an atomic op was that if it did, that would mean the kernel would have to know what values userland was using for the different mutex states, and what sort of mutex or semaphore it was dealing with (i.e. simple mutex, counting semaphore, r/w mutex etc.). It's got a lot more complex since then, and there are now circumstances where the kernel will do an atomic update to a word of user memory. Without digging through the glibc source code, I don't know exactly what circumstances they are, but I suspect it might be only when dealing with robust mutexes. Paul. |