bproc-users — General discussion about BProc.

[BProc] getpwuid and getgrgid bombing

[Daniel W. <wid...@ci...>]

#include <stdio.h>
#include <sys/bproc.h>
#include <pwd.h>
#include <sys/types.h>

main() {
        struct passwd *pwent = getpwuid(0);
        printf("%s\n",pwent->pw_name);
}


# bpsh 0 ./a.out
bpsh: Child process exited abnormally.


Same for getgrgid().  This is the problem to which Nic Henke was referring on
5/10 on this mailing list.  Apologies if this bug is already known and
listed.  The reason we're using this: pam_bproc module for access control.


Any ideas?  How do I go about debugging bpsh calls in general?  strace output
on master doesn't seem promising.  strace -f gives:

[root@alpha pam_bproc]# strace -f -o bpsh.out bpsh 0 ./a.out
Process 9229 attached
Process 9228 suspended

and bpsh.out ends in:

[pid  8940] open("/etc/ld.so.cache", O_RDONLY) = 3
[pid  8940] fstat64(3, {st_mode=S_IFREG|0644, st_size=16199, ...}) = 0
[pid  8940] old_mmap(NULL, 16199, PROT_READ, MAP_PRIVATE, 3, 0) = 0x40018000
[pid  8940] close(3)                    = 0
[pid  8940] open("/lib/libnss_files.so.2", O_RDONLY) = 3
[pid  8940] read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\200\0"..., 1024) = 1024
[pid  8940] fstat64(3, {st_mode=S_IFREG|0755, st_size=261460, ...}) = 0
[pid  8940] old_mmap(NULL, 42408, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x4015b000
[pid  8940] mprotect(0x40165000, 1448, PROT_NONE) = 0
[pid  8940] old_mmap(0x40165000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x9000) = 0x40165000
[pid  8940] close(3)                    = 0
[pid  8940] munmap(0x40018000, 16199)   = 0
[pid  8940] --- SIGSEGV (Segmentation fault) ---
[pid  8938] <... select resumed> )      = 2 (in [4 5], left {299, 640000})
[pid  8938] rt_sigprocmask(SIG_BLOCK, [CHLD], NULL, 8) = 0
[pid  8938] close(4)                    = 0
[pid  8938] read(5, "", 4096)           = 0
[pid  8938] close(5)                    = 0
[pid  8938] rt_sigprocmask(SIG_UNBLOCK, [CHLD], NULL, 8) = 0
[pid  8938] select(7, [0 3 6], [], NULL, {300, 0}) = 1 (in [6], left {300, 0})
[pid  8938] rt_sigprocmask(SIG_BLOCK, [CHLD], NULL, 8) = 0
[pid  8938] read(6, "", 4096)           = 0
[pid  8938] close(6)                    = 0
[pid  8938] wait4(-1,  <unfinished ...>
[pid  8940] +++ killed by SIGSEGV +++


Thanks,
Dan W.

-- 
-- Daniel Widyono                      http://www.cis.upenn.edu/~widyono
-- Linux Cluster Group,   CIS Dept.,  SEAS,   University of Pennsylvania
-- Mail: Rm 556, CIS Dept   200 S 33rd St         Philadelphia, PA 19104

Re: [BProc] Clustermatic

[Erik A. H. <er...@he...>]

On Wed, May 29, 2002 at 03:00:16PM +0100, Miguel Costa wrote:
> Hello.
> I've just subscribed to this mailing list and I'm sorry if this is the
> worng place to ask this but a glance through the archives suggest my
> problem is on-topic.
> 
> I've been using Scyld for a few months and just began trying out
> clustermatic. I already had redhat 7.2 installed on a separate partition
> on the master so I installed Clustermatic according to the instructions
> in README, which went fine.
> 
> The problem is that when I boot a node from the CD or a floppy
> (according to README), it hangs, I believe, when starting to download
> the phase 2 image. It sends RARP requests, retrieves the right IP
> address for its MAC address and hangs on the following line, something
> like
> 
> RARP: bproc=2223; (something I forgot, sorry);
> file=/var/beowulf/boot.img

Your problem may be the network.  Some network switches don't pass
multicast traffic fast enough for the boot image download to work.
The boot file download works with this.  If you say "mcastbcast ethX"
in /etc/beowulf/config, it will tell beoserv to use broadcast instead
of multicast.  I haven't run into a switch that couldn't handle a lot
of broadcast traffic.  In general, I'd say as switches get fancier and
more expensive they get more likely to choke on lots of multicast.

Anyway, given what you've said, that's my guess.  You can also run
beoserv with a few "-v" to see what's going on.  If you see file
requests over and over again, this is probably it.

Also if your network speed is different on the front end and the nodes
(i.e. gig-E mixed with 100mbit) you might want to say mcastthrottle
ethX YY where ethX is the interface and YY is the desired number of
megabits per second.

Improving this multicast protocol crap is somewhere on the TO-DO list.

> (I did create boot.img for the right kernel according to README)
> 
> i'm not using phase 1 from Scyld - that one, which is installed on the
> node's hard drive, doesn't even retrieve the IP adress when I boot
> clustermatic on the master

The RARP step has been modified to include some extra information.
The boot file download now uses multicast or broadcast so the Scyld
boot images and our clustermatic stuff are basically incompatible.

- Erik
-- 
Erik Arjan Hendriks          Printed On 100 Percent Recycled Electrons
er...@he...                   Contents may settle during shipment

[BProc] Clustermatic

[Miguel C. <mc...@fc...>]

Hello.
I've just subscribed to this mailing list and I'm sorry if this is the
worng place to ask this but a glance through the archives suggest my
problem is on-topic.

I've been using Scyld for a few months and just began trying out
clustermatic. I already had redhat 7.2 installed on a separate partition
on the master so I installed Clustermatic according to the instructions
in README, which went fine.

The problem is that when I boot a node from the CD or a floppy
(according to README), it hangs, I believe, when starting to download
the phase 2 image. It sends RARP requests, retrieves the right IP
address for its MAC address and hangs on the following line, something
like

RARP: bproc=2223; (something I forgot, sorry);
file=/var/beowulf/boot.img

(I did create boot.img for the right kernel according to README)

i'm not using phase 1 from Scyld - that one, which is installed on the
node's hard drive, doesn't even retrieve the IP adress when I boot
clustermatic on the master

It's weird because, from what I read in the archives from March, this
instalation seemed pretty straightforward.

any thoughts?

miguel costa

Re: [BProc] user mode linux

[Erik A. H. <er...@he...>]

On Thu, May 23, 2002 at 12:05:37AM -0400, Nicholas Henke wrote:
> Are there any plans on porting bproc to user mode linux ?

I'm not planning on it.  I can't imagine it'd be that bad though since
the per-architecture work is already done.  On the other hand, I
haven't looked at it so I really have no idea what I'm talking about.

- Erik

Re: [BProc] BProc & /dev/tty - anything to watch out?

[Erik A. H. <er...@he...>]

On Wed, May 22, 2002 at 05:56:32PM +0200, ha...@no... wrote:
> Hi, me again :-)
> 
> Seemes to be possible to let this always hold for masq-ed process:
> 
>   current->pgrp == current->bproc.masq->pgrp
>   current->session == current->bproc.masq->session
> 
> (these values therefore mostly differ from current->pid)
> 
> Looks strange at first, some basic BProc ideas seemes to be shaken,
> but it may be workable approach. Comments welcome, of course.
> 
> There are hunderds of places where pgrp/session is used in some
> way. No chance to even look at all of them.
> 
> However there are just few places where kernel compares pid to
> pgrp/session. Should be easy to fix any broken details.
> 
> I modified masq_setsid() and masq_setpgid() to also save new values to
> normal pgrp and session and things look promissing.  Now I can login
> to node via ssh without any complaints, run some commands and try to
> logout - where it hangs now (in select(), do not have details yet).
> 
> Now it also complains (on master) once per every command run from
> login bash:
> 
>    kernel: bproc: ghost(2124): failed to wait on 2132.options=2  err=-10
> 
> But I guess I would work these things out. Most of all, I would like
> to here your comments on the idea of using master's pgrp and session
> also in normal non-masq-ed process information on slave.

You're right about using the non-masq-ed being not in line with
BProc's design.  I think it should lead to some weirdness on the slave
node.  Mostly because process groups, etc are allocated with basically
the same rules as other process IDs and this will be poking in
basically random values in the slave's local process tree.

I'm surprised you're getting that error on the master though.  That
suggests that something has gotten out of sync between the slave and
the master.  Are you accumulating zombies on the front end?  Do the
process IDs (2124 & 2132) make any sense?

Having looked through the TTY code a little bit, I think the "right"
answer will be to modify it in line with other process group related
code in the kernel.  A "leader" flag will probably have to be added to
the bproc_masq_bproc_t structure.  It's not trivial though since it
likes to store PG and session IDs in its own data structures.  It
might also require allocating a PID on the slave to represent process
groups whose ID isn't already pressent on the slave.

It's gonna be messy in any case...  I need to look at this more
closely.

- Erik

[BProc] user mode linux

[Nicholas H. <he...@un...>]

Are there any plans on porting bproc to user mode linux ?

Nic

Re: [BProc] BProc & /dev/tty - anything to watch out?

[<ha...@no...>]

Hi, me again :-)

Seemes to be possible to let this always hold for masq-ed process:

  current->pgrp == current->bproc.masq->pgrp
  current->session == current->bproc.masq->session

(these values therefore mostly differ from current->pid)

Looks strange at first, some basic BProc ideas seemes to be shaken,
but it may be workable approach. Comments welcome, of course.

There are hunderds of places where pgrp/session is used in some
way. No chance to even look at all of them.

However there are just few places where kernel compares pid to
pgrp/session. Should be easy to fix any broken details.

I modified masq_setsid() and masq_setpgid() to also save new values to
normal pgrp and session and things look promissing.  Now I can login
to node via ssh without any complaints, run some commands and try to
logout - where it hangs now (in select(), do not have details yet).

Now it also complains (on master) once per every command run from
login bash:

   kernel: bproc: ghost(2124): failed to wait on 2132.options=2  err=-10

But I guess I would work these things out. Most of all, I would like
to here your comments on the idea of using master's pgrp and session
also in normal non-masq-ed process information on slave.

Best Regards

Vaclav

Re: [BProc] BProc & /dev/tty - anything to watch out?

[<ha...@no...>]

> It seemes that setsid() does not set 'leader' flag for masq-ed
> processes. This probably may be fixed in bproc-3.1.10/kernel/masq.c:
> 
>   int masq_setsid(void) {
>     ...
>     current->bproc.masq->session
> 	  = current->bproc.masq->pgrp
> 	  = current->bproc.masq->pid;
>     current->leader = 1;             /* this line added by Vaclav Hanzl */
>     return current->bproc.masq->pgrp;

This fix is not enough, also the condition testing setsid() result
from ghost is probably wrong. Seemes it should be:

int masq_setsid(void) {
    ...
    result = req->req.result;
    bproc_put_req(req);

    /*if (result == 0) {  VH: wrong test - result is session id or -1 */
    if (result > 0) {
	/* Ok, setsid succeeded for the ghost.  Now we have to do the same
	 * thing to ourselves on this side */
	current->bproc.masq->session
	    = current->bproc.masq->pgrp
	    = current->bproc.masq->pid;

	current->leader = 1; /* this line added by Vaclav Hanzl */
        
        /* VH: now result == current->bproc.masq->pid */

	return current->bproc.masq->pgrp;
    } else {
    ...


But it probably still is not enough. Now sshd seemes happy but bash
started on slave node goes to busy loop:

 --- SIGTTIN (Stopped (tty input)) ---
 rt_sigaction(SIGTTIN, {SIG_IGN}, {SIG_DFL}, 8) = 0
 ioctl(255, 0x540f, [0])                 = 0
 rt_sigaction(SIGTTIN, {SIG_DFL}, {SIG_IGN}, 8) = 0
 kill(0, SIGTTIN)                        = 0
 --- SIGTTIN (Stopped (tty input)) ---
 rt_sigaction(SIGTTIN, {SIG_IGN}, {SIG_DFL}, 8) = 0
 ioctl(255, 0x540f, [0])                 = 0
 rt_sigaction(SIGTTIN, {SIG_DFL}, {SIG_IGN}, 8) = 0
 kill(0, SIGTTIN)                        = 0

Yes, I probably stepped out to empty spaces yet to be filled by
bproc-ing ttys. Anybody cares about this or not?

Best Regards

Vaclav

Re: [BProc] BProc & /dev/tty - anything to watch out?

[<ha...@no...>]

It seemes that setsid() does not set 'leader' flag for masq-ed
processes. This probably may be fixed in bproc-3.1.10/kernel/masq.c:

  int masq_setsid(void) {
    ...
    current->bproc.masq->session
  	= current->bproc.masq->pgrp
  	= current->bproc.masq->pid;
    current->leader = 1;             /* this line added by Vaclav Hanzl */
    return current->bproc.masq->pgrp;
    ...

and following ioctl(TIOCSCTTY) then may succeed.

However this change would probably also trigger some now dormant parts
of kernel code (e.g. SIGHUP related things) which have never run for
masq-ed process so far and may either work or fail terribly.

Fortunately in kernel there are just few places where the 'leader'
flag is used:

 include/linux/sched.h:  defined here

 kernel/fork.c:          child gets 0

 kernel/sys.c:           (should be) set in sys_setsid()

 kernel/exit.c:          if set, call disassociate_ctty(1) in tty_io.c

 kernel/signal.c:        kill_sl_info() sends a signal to the session leader:
                         this is used to send SIGHUP to the controlling process
                         of a terminal when the connection is lost.
                 /* XXX bproc: Need ways to deal with session leaders...*/

 drivers/char/tty_io.c:  do_tty_hangup() sends SIGHUP&SIGCONT to leader

                         disassociate_ctty() defined here (called for leader)

                         tty_open() associates tty and leader

                         tiocsctty() works for leader only (our problem)

                         tty_ioctl(TIOCNOTTY) calls disassociate_ctty(0) for leader


I am not sure what happens when sending SIGHUP and SIGCONT, there
might even be some potential deadlock (??) but these things are beyond
my imagination - any opinions welcome. The above "XXX bproc..."
comment suggests we might step out to empty spaces...


Here is my reconstruction of things that happen before sshd fails on
ioctl(TIOCSCTTY):

1. in openssh-3.1p1/sshpty.c, pty_allocate():

  all goes well, new pty allocated for incoming ssh login

2. in openssh-3.1p1/sshpty.c, pty_make_controlling_tty():

  would like to create new session, calls setsid()

3. in linux-2.4.18-lanl.16/kernel/sys.c:

  sys_setsid(){
    ...
    bproc_hook_imr(sys_setsid,());

  process is masq-ed, so it goes to hook and will return what hook returns

4. in bproc-3.1.10/kernel/hooks.c:

  int bproc_hook_sys_setsid(void) {
    return masq_setsid();
  }

5. in bproc-3.1.10/kernel/masq.c:

  int masq_setsid(void) {
    ...
    current->bproc.masq->session
      = current->bproc.masq->pgrp
      = current->bproc.masq->pid;
    return current->bproc.masq->pgrp;

  returns WITHOUT setting current->leader = 1

6. back in openssh-3.1p1/sshpty.c, pty_make_controlling_tty():

  Wants to set controlling tty using TIOCSCTTY:

  ioctl(*ttyfd, TIOCSCTTY, NULL)

7. in linux-2.4.18-lanl.16/drivers/char/tty_io.c:

  tty_ioctl(){
    ...
    case TIOCSCTTY: return tiocsctty(tty, arg);


  tiocsctty(){
    ...
    if (!current->leader || current->tty)
      return -EPERM;

  ... current->leader should be 1 but is 0

8. in openssh-3.1p1/sshpty.c, pty_make_controlling_tty():

 sshd logs message "error: ioctl(TIOCSCTTY): Operation not permitted"

 later on in #ifdef hell it verifies once again and also logs:

 error: open /dev/tty failed - could not set controlling tty: No such file or directory


Thanks to Wilton Wong who suggested to me:

> Try putting the setgpid callbefore the setsid call.. that I think  will at
> least fix the bash error, but won't do anything about controlling tty..
> setsid() and stuff is mostly voodoo magic as fas as I am concernerd.. 

I tested this. When setpgid(0, getpid()) is before setsid() in sshd,
bash starts without complaints (and without Ctrl-C functionality),
sshd is happy with setpgid(0, getpid()) and setsid() now failes:

 n1 sshd[21367]: error: setsid: Operation not permitted 
 n1 sshd[21367]: error: ioctl(TIOCSCTTY): Operation not permitted 
 n1 sshd[21367]: error: open /dev/tty failed - could not set controlling tty: No such file or directory 

This drew my attention to strange fact that setsid() should set leader
flag and ioctl(TIOCSCTTY) does not see it.

Thanks also to Erik Arjan Hendriks for his comments.

DISCLAIMER: I did NOT try to run kernel with the proposed fix in
bproc. There may be more strange things to come.

Best Regards

Vaclav Hanzl

Re: [BProc] BProc & /dev/tty - anything to watch out?

[Wilton W. <ww...@ha...>]

Try putting the setgpid callbefore the setsid call.. that I think  will at
least fix the bash error, but won't do anything about controlling tty..
setsid() and stuff is mostly voodoo magic as fas as I am concernerd.. 

- Wilton

On Thu, 16 May 2002, Erik Arjan Hendriks wrote:

> On Tue, May 14, 2002 at 05:30:22PM +0200, ha...@no... wrote:
> > Dear BProc users and gurus,
> > 
> > I am trying to setup occasional master->slave OpenSSH access (just for
> > the convenience of environment, pty etc. during service work on slave
> > node).
> > 
> > It works but it cannot use controlling tty. Server process complains:
> > 
> >  n1 sshd[19675]: error: ioctl(TIOCSCTTY): Operation not permitted 
> >  n1 sshd[19675]: error: open /dev/tty failed - could not set controlling tty: No such device or address 
> > 
> > and client process complains:
> > 
> >  bash: initialize_job_control: setpgid: Operation not permitted
> > 
> > Then it works OK, just job control is missing. Server process is able
> > to use /dev/ptmx to allocate /dev/pts/0, it opens /dev/tty but then it
> > is unable to use it. This seemes strange to me.
> > 
> > At this moment I am not sure whether I should continue looking for
> > missing files, kernel modules etc. on slave node (usual game when you
> > insist on running some daemon :-) or delve into kernel sources and
> > look for strange interactions of BProc and controlling tty.
> > 
> > So, my only question is: Is it likely that controlling terminal things
> > are a bit broken in BProc patched kernel?
> 
> I think the odds of some of the tty stuff being busted are very good.
> The TTY code is another chunk of kernel code which deals in process
> groups that I haven't modified for BProc.
> 
> The setpgid call *should* work as usual but it's possible that it's
> getting screwed up by earlier errors or it's just broken.  The process
> group rules are pretty arcane and I don't know exactly how they're
> supposed to work.
> 
> > Just give me some opinions to let me search in more likely
> > directions...
> 
> Personally, I'd hunt down the TTY issue before worrying about the
> process group thing.  There's almost certainly some brokeness in
> there.
> 
> - Erik
> -- 
> Erik Arjan Hendriks          Printed On 100 Percent Recycled Electrons
> er...@he...                   Contents may settle during shipment
> 
> _______________________________________________________________
> 
> Have big pipes? SourceForge.net is looking for download mirrors. We supply
> the hardware. You get the recognition. Email Us: ban...@so...
> _______________________________________________
> BProc-users mailing list
> BPr...@li...
> https://lists.sourceforge.net/lists/listinfo/bproc-users

 ----[   Wilton William Wong    ]---------------------------------------------
   11060-166 Avenue         Ph : 01-780-456-9771       High Performance UNIX
   Edmonton, Alberta        FAX: 01-780-456-9772        and Linux Solutions
   T5X 1Y3, Canada                   URL: http://www.harddata.com
 -------------------------------------------------------[ Hard Data Ltd. ]----

Re: [BProc] BProc & /dev/tty - anything to watch out?

[Erik A. H. <er...@he...>]

On Tue, May 14, 2002 at 05:30:22PM +0200, ha...@no... wrote:
> Dear BProc users and gurus,
> 
> I am trying to setup occasional master->slave OpenSSH access (just for
> the convenience of environment, pty etc. during service work on slave
> node).
> 
> It works but it cannot use controlling tty. Server process complains:
> 
>  n1 sshd[19675]: error: ioctl(TIOCSCTTY): Operation not permitted 
>  n1 sshd[19675]: error: open /dev/tty failed - could not set controlling tty: No such device or address 
> 
> and client process complains:
> 
>  bash: initialize_job_control: setpgid: Operation not permitted
> 
> Then it works OK, just job control is missing. Server process is able
> to use /dev/ptmx to allocate /dev/pts/0, it opens /dev/tty but then it
> is unable to use it. This seemes strange to me.
> 
> At this moment I am not sure whether I should continue looking for
> missing files, kernel modules etc. on slave node (usual game when you
> insist on running some daemon :-) or delve into kernel sources and
> look for strange interactions of BProc and controlling tty.
> 
> So, my only question is: Is it likely that controlling terminal things
> are a bit broken in BProc patched kernel?

I think the odds of some of the tty stuff being busted are very good.
The TTY code is another chunk of kernel code which deals in process
groups that I haven't modified for BProc.

The setpgid call *should* work as usual but it's possible that it's
getting screwed up by earlier errors or it's just broken.  The process
group rules are pretty arcane and I don't know exactly how they're
supposed to work.

> Just give me some opinions to let me search in more likely
> directions...

Personally, I'd hunt down the TTY issue before worrying about the
process group thing.  There's almost certainly some brokeness in
there.

- Erik
-- 
Erik Arjan Hendriks          Printed On 100 Percent Recycled Electrons
er...@he...                   Contents may settle during shipment

Re: [BProc] BProc & /dev/tty - anything to watch out?

[<ha...@no...>]

> In the mean time you could probably get away with adding a setgpid(0, getpid())
> in to openssh source where it opens the controlling tty, I suspect this will
> fix bash but won't do squat for TIOCSCTTY..
> 
> - Wilton

I recompiled sshd with this change (supposing you ment setpgid(), not
setgpid() ?):

openssh-3.1p1, sshpty.c:

  /* ADDED BY VACLAV HANZL */
  debug("Trying to avoid bproc problems with TIOCSCTTY by setpgid(0, getpid()).");
  if (setpgid(0, getpid()) < 0)
          error("setpgid(0, getpid()): %.100s", strerror(errno));
  /* END OF PART ADDED BY VACLAV HANZL */

  debug("Setting controlling tty using TIOCSCTTY.");
  if (ioctl(*ttyfd, TIOCSCTTY, NULL) < 0)
  	error("ioctl(TIOCSCTTY): %.100s", strerror(errno));

Unfortunately, setpgid() fails as well:

 n1 sshd[19998]: error: setpgid(0, getpid()): Operation not permitted 
 n1 sshd[19998]: error: ioctl(TIOCSCTTY): Operation not permitted 

However now I can easily add any debug prints to sshd. Suggestions for
things to test are welcome.


I guess TIOCSCTTY fails because kernel does not consider the process
to be a session leader:

in kernel, drivers/char/tty_io.c:

  case TIOCSCTTY:
    return tiocsctty(tty, arg);

  int tiocsctty(...)
    ...

  static int tiocsctty(struct tty_struct *tty, int arg)
  {
    if (current->leader &&
        (current->session == tty->session))
            return 0;
    /*
     * The process must be a session leader and
     * not have a controlling tty already.
     */
    if (!current->leader || current->tty)
                return -EPERM;


Any hints welcome, I continue my effort on this...

Best Regards

Vaclav


On Tue, 14 May 2002, ha...@no... wrote:

> Dear BProc users and gurus,
> 
> I am trying to setup occasional master->slave OpenSSH access (just for
> the convenience of environment, pty etc. during service work on slave
> node).
> 
> It works but it cannot use controlling tty. Server process complains:
> 
>  n1 sshd[19675]: error: ioctl(TIOCSCTTY): Operation not permitted 
>  n1 sshd[19675]: error: open /dev/tty failed - could not set controlling tty: No such device or address 
> 
> and client process complains:
> 
>  bash: initialize_job_control: setpgid: Operation not permitted
> 
> Then it works OK, just job control is missing. Server process is able
> to use /dev/ptmx to allocate /dev/pts/0, it opens /dev/tty but then it
> is unable to use it. This seemes strange to me.
> 
> At this moment I am not sure whether I should continue looking for
> missing files, kernel modules etc. on slave node (usual game when you
> insist on running some daemon :-) or delve into kernel sources and
> look for strange interactions of BProc and controlling tty.
> 
> 
> So, my only question is: Is it likely that controlling terminal things
> are a bit broken in BProc patched kernel?
> 
> Just give me some opinions to let me search in more likely
> directions...
> 
> Thanks
> 
> Vaclav Hanzl

[BProc] BProc & /dev/tty - anything to watch out?

[<ha...@no...>]

Dear BProc users and gurus,

I am trying to setup occasional master->slave OpenSSH access (just for
the convenience of environment, pty etc. during service work on slave
node).

It works but it cannot use controlling tty. Server process complains:

 n1 sshd[19675]: error: ioctl(TIOCSCTTY): Operation not permitted 
 n1 sshd[19675]: error: open /dev/tty failed - could not set controlling tty: No such device or address 

and client process complains:

 bash: initialize_job_control: setpgid: Operation not permitted

Then it works OK, just job control is missing. Server process is able
to use /dev/ptmx to allocate /dev/pts/0, it opens /dev/tty but then it
is unable to use it. This seemes strange to me.

At this moment I am not sure whether I should continue looking for
missing files, kernel modules etc. on slave node (usual game when you
insist on running some daemon :-) or delve into kernel sources and
look for strange interactions of BProc and controlling tty.


So, my only question is: Is it likely that controlling terminal things
are a bit broken in BProc patched kernel?

Just give me some opinions to let me search in more likely
directions...

Thanks

Vaclav Hanzl

[BProc] Re: bpsh not working properly? (T#69000)

[Nicholas H. <hen...@un...>]

On Thursday 09 May 2002 05:10 pm, ygan wrote:
> Dear Admins,
> There seem to be some strange behaviour of "bpsh".
>
> When I ran ps or ls -l commands through bpsh after I reserve nodes, I
> get  the following error:
> bpsh: Child process exited abnormally.
> However the ls and tail commands run fine.
>
> Also ssh to the nodes does not work.
>
> Can you please help? Thanks!
>
> Yongchang
> PCBI
The behavior gets stranger... apparently ls is dumping core. Here is the =
bt=20
from gdb on the core file.
[henken@alpha pybproc]$ gdb -c core /bin/ls
GNU gdb Red Hat Linux (5.1-1)
Copyright 2001 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you =
are
welcome to change it and/or distribute copies of it under certain conditi=
ons.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for detail=
s.
This GDB was configured as "i386-redhat-linux"...(no debugging symbols=20
found)...
Core was generated by `ls -la'.
Program terminated with signal 11, Segmentation fault.
Reading symbols from /lib/libtermcap.so.2...(no debugging symbols=20
found)...done.
Loaded symbols for /lib/libtermcap.so.2
Reading symbols from /lib/i686/libc.so.6...done.
Loaded symbols for /lib/i686/libc.so.6
Reading symbols from /lib/ld-linux.so.2...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /lib/libnss_files.so.2...done.
Loaded symbols for /lib/libnss_files.so.2
#0  _dl_lookup_symbol (undef_name=3D0xbffff600 "_nss_files_getpwuid_r",=20
undef_map=3D0x80597f0, ref=3D0xbffff5dc, symbol_scope=3D0x805999c, type_c=
lass=3D0,
    explicit=3D1) at do-lookup.h:36
36      do-lookup.h: No such file or directory.
        in do-lookup.h
(gdb) bt
#0  _dl_lookup_symbol (undef_name=3D0xbffff600 "_nss_files_getpwuid_r",=20
undef_map=3D0x80597f0, ref=3D0xbffff5dc, symbol_scope=3D0x805999c, type_c=
lass=3D0,
    explicit=3D1) at do-lookup.h:36
#1  0x4013a108 in _dl_close (_map=3D0xbffff4b8) at dl-close.c:92
#2  0x4000d7c3 in _dl_catch_error (objname=3D0xbffff5c8, errstring=3D0xbf=
fff5cc,=20
operate=3D0x4013a0d0 <_dl_close+268>, args=3D0xbffff5d0) at dl-error.c:15=
2
#3  0x40139fed in _dl_close (_map=3D0x10) at dl-close.c:51
#4  0x4011b326 in _res_hconf_trim_domains (hp=3D0x8059178) at=20
=2E./sysdeps/i386/i486/bits/string.h:539
#5  0x4011bf56 in __nss_lookup_function (ni=3D0xbffff6e4, fct_name=3D0x40=
14df5b "s=20
down") at nsswitch.c:345
#6  0x4011d04c in __nss_hostname_digits_dots (name=3D0xbffff6e4=20
"x\221\005\b\b=F7=FF=BFTa\025@", resbuf=3D0x4014df5b, buffer=3D0xbffff6e8=
,=20
buffer_size=3D0x400,
    buflen=3D1075136032, result=3D0x8059178, status=3D0xbffff708, typep=3D=
0x40156154,=20
flags=3D1024, afp=3D0x8055d20, h_errnop=3D0xbffff738) at digits_dots.c:72
#7  0x400da147 in fgetpwent (stream=3D0x6c0b) at fgetpwent.c:77
#8  0x400d9abb in __old_getgrgid_r (gid=3D27659, resbuf=3D0x8055d20,=20
buffer=3D0xbffff768 "=A8=F9=FF=BFM=C0\004\b\013l", buflen=3D1074265511, r=
esult=3D0xbffff860)
    at getgrgid_r.c:238
#9  0x0804e2f4 in strcpy ()
#10 0x0804c04d in strcpy ()
#11 0x0804bda3 in strcpy ()
#12 0x0804ae9c in strcpy ()
#13 0x08049a97 in strcpy ()
#14 0x4003c507 in init (argc=3D134518704, argv=3D0x2, envp=3D0xbffffac4) =
at=20
=2E./sysdeps/unix/sysv/linux/init-first.c:68
(gdb)
Any ideas on this one?

--=20
Nicholas Henke
Undergraduate - Engineering 2002
--
Senior Architect and Developer
Liniac Project - University of Pennsylvania
http://clubmask.sourceforge.net
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Re: [BProc] Re: bproc+autofs: oz_pgrp problem

[<ha...@no...>]

In your last email, you already commented version of the patch which I
am yet going to write in this email :-) Good. Reverse-engineering
your comment:

Yet better way to fix autofs probably is to do just one change to
fs/autofs/autofs_i.h:

  static inline int autofs_oz_mode(struct autofs_sb_info *sbi) {
  /*	return sbi->catatonic || current->pgrp == sbi->oz_pgrp;
   * Changed to let it work with bproc (should there be any):
   */
     return sbi->catatonic || sys_getpgrp() == sbi->oz_pgrp;
  }

This should fix autofs with bproc but also compile and work without bproc.

sys_getpgrp() either contains current->pgrp in vanilla kernel or bproc
hook in bproc-patched kernel.

Regards

Vaclav


> From: Erik Arjan Hendriks <er...@he...>
> To: ha...@no...
> Cc: bpr...@li..., be...@be...
> Date: Thu, 9 May 2002 13:14:36 -0600
> User-Agent: Mutt/1.2.5.1i
> 
> On Thu, May 09, 2002 at 08:19:07PM +0200, ha...@no... wrote:
> > ...
> > #include <asm/uaccess.h>
> > 
> > /* Added by VH for bproc: */
> > #include <linux/bproc.h>
> > ...
> > 
> > static inline int autofs_oz_mode(struct autofs_sb_info *sbi) {
> > /*	return sbi->catatonic || current->pgrp == sbi->oz_pgrp;
> >  * Changed by Vaclav Hanzl to let it work with bproc:
> >  */
> >    return sbi->catatonic ||
> >      bproc_hook_imv(current->pgrp,sys_getpgrp,()) == sbi->oz_pgrp;
> > }
> > 
> > I have little idea what it actually means, but it works :)
> > Any comments welcome, as always.
> 
> That calls: (kernel/sys.c from Linux source)
> 
> asmlinkage long sys_getpgrp(void)
> {
> 	  /* SMP - assuming writes are word atomic this is fine */
> 	  return bproc_hook_imv(current->pgrp,sys_getpgrp,());
> }
> 
> the bproc_hook_imv macro calls: (kernel/hooks.c from BProc source)
> 
> int bproc_hook_sys_getpgrp(void) {
>     return current->bproc.masq->pgrp;
> }
> 
> ... but only if the caller has masqueraded PIDs which is exactly what
> you want.  This patch should perfectly safe.  It's small and to the
> point - I like it.
> ...

Re: [BProc] Re: bproc+autofs: oz_pgrp problem

[Erik A. H. <er...@he...>]

On Thu, May 09, 2002 at 08:19:07PM +0200, ha...@no... wrote:
> ...
> #include <asm/uaccess.h>
> 
> /* Added by VH for bproc: */
> #include <linux/bproc.h>
> ...
> 
> static inline int autofs_oz_mode(struct autofs_sb_info *sbi) {
> /*	return sbi->catatonic || current->pgrp == sbi->oz_pgrp;
>  * Changed by Vaclav Hanzl to let it work with bproc:
>  */
>    return sbi->catatonic ||
>      bproc_hook_imv(current->pgrp,sys_getpgrp,()) == sbi->oz_pgrp;
> }
> 
> I have little idea what it actually means, but it works :)
> Any comments welcome, as always.

That calls: (kernel/sys.c from Linux source)

asmlinkage long sys_getpgrp(void)
{
        /* SMP - assuming writes are word atomic this is fine */
        return bproc_hook_imv(current->pgrp,sys_getpgrp,());
}

the bproc_hook_imv macro calls: (kernel/hooks.c from BProc source)

int bproc_hook_sys_getpgrp(void) {
    return current->bproc.masq->pgrp;
}

... but only if the caller has masqueraded PIDs which is exactly what
you want.  This patch should perfectly safe.  It's small and to the
point - I like it.


> >This is the easiest way to get around the problem although there are
> >some drawbacks.  I've been pretty vehement about not providing hooks
> >for things like daemons to escape.  The rationale there is basically
> >that people would use those hooks if they were there. :)
> 
> Good rationale. Keep it. :-))
> 
> Little side-effect probably is that most people are not able to run
> any daemons on bproc node. :-) But things are getting better...

I don't like daemons on nodes anyway.  I've been trying for years to
reduce the bulk of stuff running on the nodes.

- Erik

Re: [BProc] Re: bproc+autofs: oz_pgrp problem

[<ha...@no...>]

> >     static inline int autofs_oz_mode(struct autofs_sb_info *sbi) {
> >       return sbi->catatonic || current->pgrp == sbi->oz_pgrp;
> >     }
> >     ...
> 
> That should work.  bproc.masq->pgrp is the value returned by getpgrp.
> Make sure you check to make sure that current->bproc.masq is not null
> though.  It's null for all non-remotely managed processes.
> 
> - Erik

I tried this but things are probably different in older bproc used in
linux-2.4.18-lanl.16 which I wanted to fix. However I got working
autofs on node with this in fs/autofs/autofs_i.h:

...
#include <asm/uaccess.h>

/* Added by VH for bproc: */
#include <linux/bproc.h>
...

static inline int autofs_oz_mode(struct autofs_sb_info *sbi) {
/*	return sbi->catatonic || current->pgrp == sbi->oz_pgrp;
 * Changed by Vaclav Hanzl to let it work with bproc:
 */
   return sbi->catatonic ||
     bproc_hook_imv(current->pgrp,sys_getpgrp,()) == sbi->oz_pgrp;
}


I have little idea what it actually means, but it works :)
Any comments welcome, as always.


More exactly, I did this:

- edited /usr/src/linux-2.4.18-lanl.16/fs/autofs/autofs_i.h as
  indicated above

- make xconfig, load configs/kernel-2.4.18-i686.config, save without changes

- make dep

- make modules

- insmod --node 1 -f /usr/src/linux-2.4.18-lanl.16/fs/autofs/autofs.o


Any lover of risky games can take my autofs.o and autofs_i.h here:

  http://noel.feld.cvut.cz/magi/bproc-autofs

and use it at HIS/HER OWN BIG RISK.

>> - Start automount outside the distributed PID space. I am not sure how
>>   to do this, bproc is damn good in not letting you escape :-) - we
>>   could modify /etc/inittab on node and signal init process and have our
>>   automount process created this way?
>
>This is the easiest way to get around the problem although there are
>some drawbacks.  I've been pretty vehement about not providing hooks
>for things like daemons to escape.  The rationale there is basically
>that people would use those hooks if they were there. :)

Good rationale. Keep it. :-))

Little side-effect probably is that most people are not able to run
any daemons on bproc node. :-) But things are getting better...


Regards

Vaclav

Re: [BProc] Re: bproc+autofs: oz_pgrp problem

[Erik A. H. <er...@he...>]

On Thu, May 09, 2002 at 04:08:16PM +0200, ha...@no... wrote:
> Starring at bproc sources, it seemes to me that autofs might be fixed
> by changing "current->pgrp" to "current->bproc.masq->pgrp" in
> fs/autofs/autofs_i.h:
> 
>     /* autofs_oz_mode(): do we see the man behind the curtain?  (The
>        processes which do manipulations for us in user space sees the raw
>        filesystem without "magic".) */
>     
>     static inline int autofs_oz_mode(struct autofs_sb_info *sbi) {
>       return sbi->catatonic || current->pgrp == sbi->oz_pgrp;
>     }
> 
> Please comment - does it make sense? Is current->bproc.masq->pgrp the
> lie which is returned to node process calling getpgrp() ?

That should work.  bproc.masq->pgrp is the value returned by getpgrp.
Make sure you check to make sure that current->bproc.masq is not null
though.  It's null for all non-remotely managed processes.

- Erik

Re: [BProc] bproc+autofs: oz_pgrp problem

[Erik A. H. <er...@he...>]

On Thu, May 09, 2002 at 12:22:15PM +0200, ha...@no... wrote:
> I may be wrong with my analysis, I am no expert on any of the things
> involved (bproc, autofs). Please correct me if I am wrong.

I'm not intimately familiar with autofs but your analysis sounds
dead-on to me.

> If I am right, there are several possible ways to avoid deadlock:
> 
> - Make modified autofs.o which is aware of bproc-related process group
>   tricks - autofs_oz_mode() should test for the same value which is
>   returned via getpgrp() (hope this can avoid node-head-node
>   communication). We should also verify whether setpgrp() used in
>   automount.c would work as expected.

This would work but it'd be somewhat unpleasant to actually do.  You'd
have suck part of bproc-x.x.x/kernel/bproc.h into the kernel source to
get the defintions for struct bproc_masq_proc_t.

> - Start automount outside the distributed PID space. I am not sure how
>   to do this, bproc is damn good in not letting you escape :-) - we
>   could modify /etc/inittab on node and signal init process and have our
>   automount process created this way?

This is the easiest way to get around the problem although there are
some drawbacks.  I've been pretty vehement about not providing hooks
for things like daemons to escape.  The rationale there is basically
that people would use those hooks if they were there. :)

> Any opinions and suggestions are more than welcome, especially
> comments on current handling of process groups in bproc (I know
> nothing about it).

The kernel lies to user process about process groups pretty much the
same way as with pids.

I don't have a nice solution handy for this one right now.

- Erik

[BProc] Re: bproc+autofs: oz_pgrp problem

[<ha...@no...>]

Starring at bproc sources, it seemes to me that autofs might be fixed
by changing "current->pgrp" to "current->bproc.masq->pgrp" in
fs/autofs/autofs_i.h:

    /* autofs_oz_mode(): do we see the man behind the curtain?  (The
       processes which do manipulations for us in user space sees the raw
       filesystem without "magic".) */
    
    static inline int autofs_oz_mode(struct autofs_sb_info *sbi) {
      return sbi->catatonic || current->pgrp == sbi->oz_pgrp;
    }

Please comment - does it make sense? Is current->bproc.masq->pgrp the
lie which is returned to node process calling getpgrp() ?

Vaclav

[BProc] bproc+autofs: oz_pgrp problem

[<ha...@no...>]

Hi,

I investigated problems with autofs on bproc node and I believe there
is deadlock caused by interference of the way bproc handles process
groups and autofs use them. 

Here is a minimalistic way to invoke the problem:

  modprobe --node 1 autofs
  bpsh 1 mkdir -p /usr/lib/autofs
  for x in /usr/lib/autofs/*; do bpcp $x 1:$x; done
  bpsh 1 touch /etc/yyy
  bpsh 1 automount /xxx file /etc/yyy
  bpsh 1 ps|grep 'automount'

  ...  22113 ?        00:00:00 automount

  bpsh 1 strace -p 22113

And in another window:

  bpsh 1 ls /xxx/zzz

If you also have this problem, ls hangs, only -9 kills automount and
strace prints:

  read(4, "\3\0\0\0\0\0\0\0\2\0\0\0\3\0\0\0zzz\0\0\0\0\0\0\0\0\0\0"..., 272) = 272
  chdir("/xxx")                           = 0
  lstat64("zzz", 

With a lot of guesswork (as I am no expert in this) I deduced this
scenario:

1. automount process finds out its own process group using getpgrp()

2. it mounts itself, using mount executable with pgrp option set to
   the result of getpgrp()

3. kernel performs mount and stores pgrp as oz_pgrp: processes with
   this 'magic' pgrp can see raw directories instead of automounted
   ones

4. when automounted subdirectory is accessed, kernel writes pipe to
   automount asking it to mount what should be seen there

5. automount tests whether automounted subdirectory exists; as it
   belongs to oz_pgrp, it should see raw directories

6. kernel fails to recognize that automount belongs to oz_pgrp and
   instead of showing raw directories, it wants to ask automount
   itself to deliver what should be seen. Deadlock.

I believe that the problem is that getpgrp() on bproc node returns
something else than current->pgrp tested inside the kernel on bproc
node in autofs_oz_mode().

In more details, the scenario looks like this:

1. in autofs-4.0.0pre10/daemon/automount.c:

   /* Make our own process group for "magic" reason: processes that share
      our pgrp see the raw filesystem behine the magic.  So if we are a 
      submount, don't change -- otherwise we won't be able to actually
      perform the mount.  A pgrp is also useful for controlling all the
      child processes we generate. */
   if ( !submount && setpgrp() ) {
     syslog(LOG_CRIT, "setpgrp: %m");
     exit(1);
   }
   my_pgrp = getpgrp();

 (I am not sure whether setpgrp() is called or not but I think it
 might not matter now. It may however cause more problems beyond the
 one described here.)

2. still in autofs-4.0.0pre10/daemon/automount.c, in mount_autofs(...):

    sprintf(options, "fd=%d,pgrp=%u,minproto=2,maxproto=%d", pipefd[1],
  	    (unsigned)my_pgrp, AUTOFS_MAX_PROTO_VERSION);
    sprintf(our_name, "automount(pid%u)", (unsigned)my_pid);
    
    if (spawnl(LOG_DEBUG, PATH_MOUNT, PATH_MOUNT, "-t", "autofs", "-o",
  	       options, our_name, path, NULL) != 0) {
      syslog(LOG_CRIT, "cannot find autofs in kernel");

3. in kernel in linux/fs/autofs/inode.c, autofs_read_super():

    if ( parse_options(data,&pipefd,&root_inode->i_uid,&root_inode->i_gid,&s
      bi->oz_pgrp,&minproto,&maxproto) ) {
        printk("autofs: called with bogus options\n");
        goto fail_dput;
    }

  ("magic" group got to oz_pgrp)

4. in kernel in system call invoked by ls:

  kernel writes pipe to automount process, asking it to arrange things
  in /xxx/zzz. System call does not return to ls until automount does
  its work (and therefore does never return).

5. in autofs-4.0.0pre10/daemon/automount.c:

   static int handle_packet_missing(...)
     ...
     chdir(ap.path);
     if ( lstat(pkt->name,&st) == -1 ||
  	  (S_ISDIR(st.st_mode) && st.st_dev == ap.dev) ) {
       /* Need to mount or symlink */

  (lstat() should see raw directories, but will not and will hang)

6. in kernel in fs/autofs/autofs_i.h:

    /* autofs_oz_mode(): do we see the man behind the curtain?  (The
       processes which do manipulations for us in user space sees the raw
       filesystem without "magic".) */
    
    static inline int autofs_oz_mode(struct autofs_sb_info *sbi) {
      return sbi->catatonic || current->pgrp == sbi->oz_pgrp;
    }

  (automount should be recognized as "magic" with autofs_oz_mode()==1
   but instead gets the same treatment as ls in step 4. above. There
   is probably kernel lock around autofs things to happen, or maybe
   kernel even sends request to autofs via pipe - in any case it have to
   deadlock as automount still waits for lstat())

*******************************

I may be wrong with my analysis, I am no expert on any of the things
involved (bproc, autofs). Please correct me if I am wrong.

If I am right, there are several possible ways to avoid deadlock:

- Make modified autofs.o which is aware of bproc-related process group
  tricks - autofs_oz_mode() should test for the same value which is
  returned via getpgrp() (hope this can avoid node-head-node
  communication). We should also verify whether setpgrp() used in
  automount.c would work as expected.

- Start automount outside the distributed PID space. I am not sure how
  to do this, bproc is damn good in not letting you escape :-) - we
  could modify /etc/inittab on node and signal init process and have our
  automount process created this way?


Any opinions and suggestions are more than welcome, especially
comments on current handling of process groups in bproc (I know
nothing about it).

Thanks and Best Regards

Vaclav Hanzl


========= copy of my original message on beowulf maillist: ========

Subject: autofs mount on bproc node?
From: hanzl
To: be...@be...
Date: Wed, 08 May 2002 17:18:39 +0200

Hi,

any of you great gurus managed to use autofs on bproc nodes?


I am pushing hard, but at this moment it looks like hitting concrete
wall with my head... any help would be more than welcome.

My nodes already run NFS client, NFS server and syslogd/klogd.
Automount seemes to start OK but when I ls automounted directory (node
is client, head is server), ls hangs, automount process hangs (kill -9
needed to kill it) and there is no error message anywhere.

I have syslogd working on node and automount is full of syslog()
calls, but in this case is says nothing, it probably hangs early when
receiving automount request from kernel.

The same automount setup works when head is client and node is server.

(Is there any way to force compiled kernel to give out more debug
messages? E.g. write somewhere to /proc?)


Running daemons on bproc nodes is tricky and probably worth
mini-howto. (Yes, I try to avoid daemons on nodes, but sometimes I
really need them.) If you know any related documents, please let me
know.

Thanks

Vaclav

-------------------------------------------------------

My setup:

RedHat 7.2 with most rpm updates till Nov 2001, Clustermatic (March
2002 version), kernel 2.4.18-lanl.16, automount version 3.1.7 (also
tested 4.0.0).

==> /etc/auto.master <==
# /etc/auto.master
/nfs  /etc/auto.nfs  rw,intr,rsize=8192,wsize=8192

==> /etc/auto.nfs <==
# /etc/auto.nfs
*  -fstype=autofs,-Dhost=&  file:/etc/auto.sub

==> /etc/auto.sub <==
# /etc/auto.sub
*  ${host}:/&

==> /etc/beowulf/syslog.conf <==
# syslog.conf for magi nodes

# log everything on screen:
*.*                                                     /dev/console

# log everything to head (magi):
*.*                                                     @10.0.4.1

==> /etc/beowulf/exports.node <==
# exports for beowulf node (experimentel)
/etc 10.0.4.1(ro)

==> /etc/beowulf/nsswitch.conf <==
passwd: bproc files
hosts: bproc

==> /etc/exports <==
# magi exports
/bin 10.0.4.0/255.255.255.0(ro)
/home noel(rw) 10.0.4.0/255.255.255.0(ro)
/lib 10.0.4.0/255.255.255.0(ro)
/sbin 10.0.4.0/255.255.255.0(ro)
/usr 10.0.4.0/255.255.255.0(ro)
/var 10.0.4.0/255.255.255.0(ro)

==> /etc/sysconfig/syslog <==
# Options to syslogd
# -m 0 disables 'MARK' messages.
# -r enables logging from remote machines
# -x disables DNS lookups on messages recieved with -r
# See syslogd(8) for more details
#SYSLOGD_OPTIONS="-m 0"
# VH: added -r for log from magi nodes to magi master:
SYSLOGD_OPTIONS="-m 0 -r"
# Options to klogd
# -2 prints all kernel oops messages twice; once for klogd to decode, and


== And my main experimental script: ==



#!/bin/bash

# beowulf node startup for magi

# New (RH7.2_Clustermatic/magi) version

echo This is /nfs/noel/home/hanzl/beowulf/startnode

########### CONFIG AREA ############

#nodenum:
N=1

#master IP:
MASTER=10.0.4.1

# directories to NFS-replicate from master (read only):

NFSDIRS='bin sbin usr'

# directories to just create on node:

CREATEDIRS='tmp nfs .autofsck var/lib/nfs/sm var/run var/lock/subsys var/nis'

# files to copy from master to node with identical pathname:

COPYFILES='/etc/auto.master /etc/auto.nfs /etc/auto.sub /etc/services /etc/rpc /etc/protocols /etc/passwd /etc/group'

# modules needed on nodes:

MODULES='sunrpc lockd nfs nfsd autofs'

####################################

echo Master IP: $MASTER, node number: $N

bpstat $N

echo Rebooting node $N...

bpctl -S $N -s reboot

while true; do
 STATUS=`bpstat $N -s`
 echo -n ' '$STATUS
 if [[ "$STATUS" == "up" ]]; then break; fi
 sleep 1
done

echo ''
echo Node $N is up!

####################################

echo "Creating any missing directories on node $N:"

for d in $NFSDIRS $CREATEDIRS; do
  echo -n ' '$d
  bpsh $N mkdir -p /$d
done

echo ''
echo ... done

####################################

echo "Inserting modules on node $N"

for m in $MODULES; do
  echo -n ' '$m
  modprobe --node $N $m
done

echo ''
echo ... done

####################################

echo "Copying config files to node $N"

echo "Files with identical pathname:"

for f in $COPYFILES; do
  echo -n ' '$f
  bpcp $f $N:$f
done

echo ''
echo ... done

echo "Files with different pathname:"

bpcp /etc/beowulf/syslog.conf $N:/etc/syslog.conf
bpcp /etc/beowulf/exports.node $N:/etc/exports
bpcp /etc/beowulf/nsswitch.conf $N:/etc/nsswitch.conf

echo ... done

####################################




####################################
echo "Hello to console"|bpsh $N dd of=/dev/console 2>/dev/null

####################################


echo "Starting basic daemons on node $N"


bpsh $N touch /var/lib/nfs/rmtab 
bpsh $N touch /var/lib/nfs/xtab  
bpsh $N touch /var/lib/nfs/etab

echo "portmap..."
bpsh $N portmap

echo "syslogd..."
bpsh $N syslogd

echo "klogd..."
bpsh $N klogd

bpsh $N logger "Starting basic rpc daemons on node $N"

echo "rpc.statd..."
# statd will run in /var/lib/nfs/statd as rpcuser and needs rw access
bpsh $N mkdir -p /var/lib/nfs/statd
bpsh $N chown rpcuser /var/lib/nfs/statd
bpsh $N chgrp rpcuser /var/lib/nfs/statd
bpsh $N rpc.statd 
bpsh $N touch /var/lock/subsys/nfslock

echo "Expected rpcinfo:" nlockmgr portmapper status
echo "  Actual rpcinfo:" `bpsh $N rpcinfo -p|awk '{print $5}'|sort|uniq`
echo "(maybe they did not start yet...)"
# 'nlockmgr' is provided by kernel module, not daemon

####################################
# (This could have been done even just after portmap, but is probably safer here)
echo "Mounting NFS directories, node $N is client, head is server"
bpsh $N logger "Mounting NFS directories, node $N is client, head is server"
for d in $NFSDIRS; do
  echo -n ' '$d
  bpsh 1 mount 10.0.4.1:/$d /$d
done
echo ''
echo ... done
####################################

echo "Expected rpcinfo:" nlockmgr portmapper status
echo "  Actual rpcinfo:" `bpsh $N rpcinfo -p|awk '{print $5}'|sort|uniq`



echo "Starting NFS server daemons on node $N"
bpsh $N logger "Starting NFS server daemons on node $N"

# We should provide: mountd nfs rquotad

RPCNFSDCOUNT=8

echo 'exportfs (not daemon)...'
bpsh $N exportfs -r

## echo "rpc.rquotad"
## bpsh $N rpc.rquotad
## # hangs (but shows in rpcinfo), avoid it

echo "rpc.mountd..."
# Special treatment needed: rpc.mountd with socket on stdin goes mad
# and therefore cannot be started by bpsh directly. Stdin redirect
# on node helps. You can either NFS-mount /usr/sbin or you can use
# new bproc, which delivers missing executables when absolute path
# is used.
bpsh $N sh -c '/usr/sbin/rpc.mountd </dev/null'

echo "rpc.nfsd (count=$RPCNFSDCOUNT)..."
bpsh $N rpc.nfsd $RPCNFSDCOUNT

bpsh $N touch /var/lock/subsys/nfs

echo "NFS server on node $N should work now"

####################################

echo "Expected rpcinfo:" mountd nfs nlockmgr portmapper status
echo "  Actual rpcinfo:" `bpsh $N rpcinfo -p|awk '{print $5}'|sort|uniq`

echo "Testing automount (head is client, node $N is server):"
echo "ls /nfs/n$N/etc:"
ls /nfs/n$N/etc

####################################

echo "Processes on node $N seen in head PID-space:"

ps aux|bpstat -P|grep ^$N'[^0-9]'

####################################

exit

echo "Starting autofs client on node $N"
bpsh $N logger "Starting autofs client on node $N"

bpsh $N /usr/sbin/automount /nfs file /etc/auto.nfs rw,intr,rsize=8192,wsize=8192
bpsh $N touch /var/lock/subsys/autofs

#bpsh $N sh -c '/usr/sbin/automount /nfs file /etc/auto.nfs rw,intr,rsize=8192,wsize=8192'

#bpsh $N sh -c '/root/autofs-4.0.0pre10/daemon/automount /nfs file /etc/auto.nfs rw,intr,rsize=8192,wsize=8192'

# bpsh $N ls /nfs/n-1/var wil HANG !!! :-(


==================== END =====================

Re: [BProc] bpsh patch

[Erik A. H. <er...@he...>]

On Fri, Apr 26, 2002 at 03:47:37PM -0400, Sean DIlda wrote:
> We found a bug in bpsh where if you send a single command to several
> (hundreds) of nodes at once, the output from some nodes would be lost.
> 
> It turns out that in this case, some of the remote commands would finish
> running and exit before bpsh ever started handling I/O, therefore the
> sigchld handler would decrease outstanding_connections, then when bpsh
> started handling those connections (which were still pending) it
> decreased outstanding_connections again.  This led to a condition where
> outstanding connections would be way below 0, and io_to_do and
> late_connections would both be zero, and thus bpsh would stop looping an
> exit, even though there might still be connections it hasn't handled
> yet.  The attached patch keeps outstanding_connections from being
> decremented twice for a node that finishes early, thus solving the
> problem.

The patch looks good to me.  I'll apply it.

- Erik
-- 
Erik Arjan Hendriks          Printed On 100 Percent Recycled Electrons
er...@he...                   Contents may settle during shipment

[BProc] bpsh patch

[Sean D. <ag...@sc...>]

We found a bug in bpsh where if you send a single command to several
(hundreds) of nodes at once, the output from some nodes would be lost.

It turns out that in this case, some of the remote commands would finish
running and exit before bpsh ever started handling I/O, therefore the
sigchld handler would decrease outstanding_connections, then when bpsh
started handling those connections (which were still pending) it
decreased outstanding_connections again.  This led to a condition where
outstanding connections would be way below 0, and io_to_do and
late_connections would both be zero, and thus bpsh would stop looping an
exit, even though there might still be connections it hasn't handled
yet.  The attached patch keeps outstanding_connections from being
decremented twice for a node that finishes early, thus solving the
problem.

Sean

[BProc] Beoboot patch

[Wilton W. <ww...@ha...>]

this patch adds a directive to /etc/beowulf/config and /etc/beowulf/config.boot
to allow you to specify which devices should be loaded in the the boot images
(config.boot) and which should be loaded by the node_up script (config)

probably want a minimum of at least console, kmem and zero in your images and
on your nodes.

config.boot lines look like:

<SNIP>
bootdev console kmem zero
</SNIP>

config lines look like:

<SNIP>
devlist console kmem zero
devlist hda hda1 hda2 hda3 hda4 hda5
</SNIP>

- Wilton

 ----[   Wilton William Wong    ]---------------------------------------------
   11060-166 Avenue         Ph : 01-780-456-9771       High Performance UNIX
   Edmonton, Alberta        FAX: 01-780-456-9772        and Linux Solutions
   T5X 1Y3, Canada                   URL: http://www.harddata.com
 -------------------------------------------------------[ Hard Data Ltd. ]----

[BProc] ANNOUNCE: BProc 3.1.10

[Erik A. H. <er...@he...>]

Available in the usual place:

http://sourceforge.net/project/showfiles.php?group_id=24453&release_id=86348

3.1.10 ---------------------------------------------------------------------

  This release features a much-improved vrfork and a vexecmove.  These two
  interfaces still need some work in the error cases but they seem pretty
  stable at this point.  Documentation is still a to-do here.

  There's also the usual round of bug fixes.

Changes from 3.1.9 to 3.1.10

        * Added vexecmove to do basically the same thing as vrfork with
        arbitrary binaries.

        * Rewrote vrfork (and therefore also vexecmove) so handle multiple
        processes per node somewhat reasonably.

        * Added "locked" mode for bpstat.  (-L)

        * Fixed a bug in move interface code that could lead to kernel
        oopses.