Re: [BProc] clone support

[Nicholas H. <he...@se...>]

On Fri, 2003-08-01 at 11:15, er...@he... wrote:
> Yeah, pthreads gets its dirty little fingers in a lot of places.  I
> believe it mostly just tries to clean up with its wrappers around
> fork, clone, etc.
> 
> Another unfortunate side effect of the pthreads implementation is that
> it does not recognize bproc_rfork, etc. as fork-like calls so the
> child process is sometimes hosed if those are called from a
> pthreadeded program.  The whole mess is severely flawed, IMO.
> 
> - Erik
> 
> P.S. I'm talking about Linuxthreads here, not NPTL.  I haven't looked
> at NPTL yet but it should be much less screwy.

Ok -- I have done some digging on this -- just for my sadistic self :)

So -- I started with the program (bcl) that J.A sent, and found a few
interesting things.

1) If the program is compiled with -lpthread and -static, I can actually
get past the bproc_rfork, and into nslave. The clone command is
executed, and the pslaves fail with a SIGSEGV. 

1a) If you bproc_rfork to '-1' -- it works just fine. I am assuming that
bproc_rfork in that case is just a regular fork.

2) When linked with -lpthread, pthread basically wraps all of the
syscalls with a 'CANCELABLE_SYSCALL', that trys to use
'pthread_setcanceltype' around the call -- not sure _why_, but it does.
This call, the one before the actuall syscall fails, as in the following
code:
int pthread_setcanceltype(int type, int * oldtype)
{
  pthread_descr self = thread_self();
  if (type < PTHREAD_CANCEL_DEFERRED || type >
PTHREAD_CANCEL_ASYNCHRONOUS)
    return EINVAL;
  if (oldtype != NULL) *oldtype = THREAD_GETMEM(self, p_canceltype);
  THREAD_SETMEM(self, p_canceltype, type);
  if (THREAD_GETMEM(self, p_canceled) &&
      THREAD_GETMEM(self, p_cancelstate) == PTHREAD_CANCEL_ENABLE &&
      THREAD_GETMEM(self, p_canceltype) == PTHREAD_CANCEL_ASYNCHRONOUS)
    __pthread_do_exit(PTHREAD_CANCELED, CURRENT_STACK_FRAME);
  return 0;
}

This is where the seg fault happens, in the first THREAD_GETMEM, as the
pointer 'self' points to memory that cannot be accessed. In this case,
it appears that THREAD_GETMEM is just doing a 'self->p_canceltype', or a
regular structure member dereference.

So.... it would appear that pthreads is getting confused as to where
'self' actually points. At this point, I am guessing that that
information is stored in the shared library somewhere, and that memory
is not getting dumped to the remote node. Am I correct there? If so --
would the work done on the BLCR ( checkpoint/restart ) based on vmadump
help -- I believe they have some hacks to get the pthread manager setup
correctly on the node.


Thoughts?
Nic

P.S I have a feeling this is also what happens to Java ( and my real
motivation to fix it.), as it uses clone, and is linked with -lpthread.
-- 
Nicholas Henke
Penguin Herder & Linux Cluster System Programmer
Liniac Project - Univ. of Pennsylvania