bproc-users

[BProc] Process dies a short while after bproc_move

[Julian S. <ju...@va...>]

I wrote a simple test program which simply consists of a
spin-wait loop, then a bproc_move from front end to a slave 
node, and a second spin-wait loop which prints a progress 
message every second or so.

The process is migrated correctly.  However, after running
on the slave for a few (20 ish?) seconds, it dies, with
"Killed" printed.  The amount of progress it makes before
this happens varies from attempt to attempt, although it
does not vary by much.

Another ten or twenty seconds after "Killed" appears, the
slave invariably reboots itself.

strace isn't helpful; it merely tells me the process is killed
by SIGKILL, which is apparent anyway.

Why does this happen?  How can I avoid it?  Given that the
migration takes place OK, it feels like the master has asked
the slave to reset itself as a result of some kind of timeout
happening.  When the slaves are idle they stay alive indefinitely
with no such reboots.

[I also don't understand why I can see printfs from the 
 program after migration, given that "man 2 bproc_move" says
 "All open files are closed during migration."]

--------

I'm using x86 BProc from Clustermatic 5.  The master is
vanilla SuSE 9.2.  Both the master and slave(s) run
on VMWare WS5 running on WinXP on a P4 with 1G memory.
All /etc/clustermatic/config settings are defaults, except
"pingtimeout", which I reduced from 30 to 10.

The program prints a message just before/after it does bproc_move.
I can see that the migration really does happen watching the CPU
usage of the various VMware threads involved.

J

Re: [BProc] Process dies a short while after bproc_move

[Erik H. <eah...@gm...>]

On 6/21/05, Julian Seward <ju...@va...> wrote:
>=20
> I wrote a simple test program which simply consists of a
> spin-wait loop, then a bproc_move from front end to a slave
> node, and a second spin-wait loop which prints a progress
> message every second or so.
>=20
> The process is migrated correctly.  However, after running
> on the slave for a few (20 ish?) seconds, it dies, with
> "Killed" printed.  The amount of progress it makes before
> this happens varies from attempt to attempt, although it
> does not vary by much.
>=20
> Another ten or twenty seconds after "Killed" appears, the
> slave invariably reboots itself.

When a slave holding a process dies, the process looks like it got a
SIGKILL on the front end.  There's no normal UNIX way to say something
like the machine that process was on isn't with us anymore so that
seemed like the next best thing to do.

> strace isn't helpful; it merely tells me the process is killed
> by SIGKILL, which is apparent anyway.
>=20
> Why does this happen?  How can I avoid it?  Given that the
> migration takes place OK, it feels like the master has asked
> the slave to reset itself as a result of some kind of timeout
> happening.  When the slaves are idle they stay alive indefinitely
> with no such reboots.

I'm not sure why that's happening.  Is there anything on the slave's
console?  The 20 seconds interval sounds like the normal ping timeout
scenerio between the master and a slave.  Is it possible that the
slave daemon is getting starved somehow?  Can you bpsh other things to
the node while your program is running?
=20
> [I also don't understand why I can see printfs from the
>  program after migration, given that "man 2 bproc_move" says
>  "All open files are closed during migration."]

That's an inaccuracy, I suppose.  If you don't specify any other I/O
setup, it takes stdout + stderr and feeds them to what ever the
process's original stdout was.  It uses the socket that was used to
pass the process data to do this.  It's a crutch for little test
programs like the one you wrote.  The I/O forwarding done by bpsh,
mpirun, etc. doesn't work this way.

- Erik