open-sharedroot-devel

Re: [OSR-devel] killall5 output

[Marc G. <gr...@at...>]

----- "Gordan Bobic" <go...@bo...> wrote:

> On 25/12/2009 21:16, Marc Grimme wrote:
> > On output of killall5:
> > It should output to the console. Which means you should see your
> printfs
> > on console.
> 
> That's what I thought - only it doesn't. killall5 starts, and nothing
> 
> happens after that. I wonder if the problem actually occurs before
> that, 
Try to first add an echo before killall5 is issued to see what would be called.
I often added a bash/sh call afterwards to test what's happening:
action $"Sending all processes the TERM signal..." /usr/comoonics/killall5 -15 ${KILLALL_OPTS}
sleep 5
action $"Sending all processes the KILL signal..."  /usr/comoonics/killall5 -9 ${KILLALL_OPTS}
->
echo "Sending all processes the TERM signal..." /usr/comoonics/killall5 -15 ${KILLALL_OPTS}
bash
action $"Sending all processes the TERM signal..." /usr/comoonics/killall5 -15 ${KILLALL_OPTS}
sleep 5
action $"Sending all processes the KILL signal..."  /usr/comoonics/killall5 -9 ${KILLALL_OPTS}

Then you know what happens. I don't think bootsr does any harm see below.
In the shell you could also try your command.

> e.g. the fact that init.d/bootsr does it's "Stopping fuse dependent 
> services" by calling "cc_init stop"...
> 
> Except this should resolve to fuse_init, which doesn't actually appear
> 
> to be implemented. glusterfs_init is. But interestingly, cc_init stop
> 
> takes about 30 seconds to return even though it seems to be invoking a
> 
> non-existant function.
Thats something else. It is what it takes to detect where the chroot resides.
So I wouldn't bother with this.
> 
> Having said that, it seems to do "Stopping gfs dependent services",
> too, 
> which souldn't happen on glusterfs at all since it has nothing to do 
> with gfs, so I'm not sure right now where that's coming from...
It should cause your clustertype is "gfs". So basically this happens. This is a kind of not yet implemented part. As you are using an /etc/cluster/cluster.conf the clustertype needs to be gfs.
And bootsr stops all services relevant to the rootfs (glusterfs/fuse) and clustertype (gfs). So this is normal too.
> 
> > You need to copy it to /usr/comoonics/killall5.
> 
> Yes, did that.
Thought that. Just to be sure.
> 
> > You could
> > output the killall opts to console in order to see if the programs
> get
> > excluded within killing:
> > echo "/usr/comoonics/killall5 -15 ${KILLALL_OPTS}"
>  > just before it is called and you should see if it is called
> properly.
> 
> Did that, too, and that looks fine. But killall5 never returns, and 
> never prints any output.
Never returns is suspicious. Try to call it yourself.
> 
> > Background information for the latest versions (OT but perhaps
> > interesting information for further steps):
> > Since latest bootimage versions the shutdown process has slightly
> > changed. We don't need so many patches but just call our own
> version
> > of halt.local (called in rc.sysinit). halt.local should be from the
> > bootimage rpm a symlink to
> > /sbin/halt.local -> 
> /opt/atix/comoonics-bootimage/boot-scripts/com-halt.sh.
> >
> > Nevertheless the killall5 call is issued before (if I recall it
> right)
> > so the killall5-patch needs to be inplace.
> 
> All of which still doesn't explain why killall5 goes away and silently
> 
> dies. :(
Yes.
So that's the question.

> 
> > With latest versions of bootimage you can even enable the stepmode
> in the
> > com-halt.sh (means in halt/reboot mode). There should be a skript
> called
> > /opt/atix/comoonics-bootimage/com-chroot. This helps you get
> chrooted into
> > the exitrd (former ramdisk of initrd). By default in
> /var/comoonics/chroot.
> > Then just issue a "setparameter step" and the reboot/halt will ask
> for
> > userinput after every command.
> >
> > Unfortunatly this does not help for killall5 as this is called
> before.
> 
> And even if it was applicable here, the key problem is that stepping 
> through wouldn't help. I know where it's going wrong, I just can't see
> 
> why. killall5 starts, and then silently locks up.
> 
> Gordan
> P.S.
> Again, should I CC to the list?
Yes. I'll do. I didn't see this. Thought you had.
-- 

Marc Grimme

Tel: +49 89 4523538-14
Fax: +49 89 9901766-0
E-Mail: gr...@at...

ATIX Informationstechnologie und Consulting AG | Einsteinstrasse 10 |
85716 Unterschleissheim | www.atix.de | www.open-sharedroot.org

Registergericht: Amtsgericht Muenchen, Registernummer: HRB 168930, USt.-Id.: 
DE209485962 | Vorstand: Marc Grimme, Mark Hlawatschek, Thomas Merz (Vors.) |
Vorsitzender des Aufsichtsrats: Dr. Martin Buss

Re: [OSR-devel] killall5 output

[Gordan B. <go...@bo...>]

On 26/12/2009 21:44, Marc Grimme wrote:

>>> On output of killall5:
>>> It should output to the console. Which means you should see your
>> printfs
>>> on console.
>>
>> That's what I thought - only it doesn't. killall5 starts, and nothing
>>
>> happens after that. I wonder if the problem actually occurs before
>> that,
> Try to first add an echo before killall5 is issued to see what would be called.
> I often added a bash/sh call afterwards to test what's happening:
> action $"Sending all processes the TERM signal..." /usr/comoonics/killall5 -15 ${KILLALL_OPTS}
> sleep 5
> action $"Sending all processes the KILL signal..."  /usr/comoonics/killall5 -9 ${KILLALL_OPTS}
> ->
> echo "Sending all processes the TERM signal..." /usr/comoonics/killall5 -15 ${KILLALL_OPTS}
> bash
> action $"Sending all processes the TERM signal..." /usr/comoonics/killall5 -15 ${KILLALL_OPTS}
> sleep 5
> action $"Sending all processes the KILL signal..."  /usr/comoonics/killall5 -9 ${KILLALL_OPTS}
>
> Then you know what happens. I don't think bootsr does any harm see below.
> In the shell you could also try your command.

I just did that, and the result is as expected. killall5 starts and 
never returns. No output.

>>> You could
>>> output the killall opts to console in order to see if the programs
>> get
>>> excluded within killing:
>>> echo "/usr/comoonics/killall5 -15 ${KILLALL_OPTS}"
>>   >  just before it is called and you should see if it is called
>> properly.
>>
>> Did that, too, and that looks fine. But killall5 never returns, and
>> never prints any output.
 >
> Never returns is suspicious. Try to call it yourself.

I did - and it really doesn't return.
I'm a bit stumped at this, especially since the effect is the same both 
on the original killall5 that ships with OSR sysvinit and my modified 
killall5, so I'm reasonably sure that killall5 isn't broken or miscompiled.

Gordan

Re: [OSR-devel] killall5 output

[Gordan B. <go...@bo...>]

Gordan Bobic wrote:
> On 26/12/2009 21:44, Marc Grimme wrote:
> 
>>>> On output of killall5:
>>>> It should output to the console. Which means you should see your
>>> printfs
>>>> on console.
>>> That's what I thought - only it doesn't. killall5 starts, and nothing
>>>
>>> happens after that. I wonder if the problem actually occurs before
>>> that,
>> Try to first add an echo before killall5 is issued to see what would be called.
>> I often added a bash/sh call afterwards to test what's happening:
>> action $"Sending all processes the TERM signal..." /usr/comoonics/killall5 -15 ${KILLALL_OPTS}
>> sleep 5
>> action $"Sending all processes the KILL signal..."  /usr/comoonics/killall5 -9 ${KILLALL_OPTS}
>> ->
>> echo "Sending all processes the TERM signal..." /usr/comoonics/killall5 -15 ${KILLALL_OPTS}
>> bash
>> action $"Sending all processes the TERM signal..." /usr/comoonics/killall5 -15 ${KILLALL_OPTS}
>> sleep 5
>> action $"Sending all processes the KILL signal..."  /usr/comoonics/killall5 -9 ${KILLALL_OPTS}
>>
>> Then you know what happens. I don't think bootsr does any harm see below.
>> In the shell you could also try your command.
> 
> I just did that, and the result is as expected. killall5 starts and 
> never returns. No output.
> 
>>>> You could
>>>> output the killall opts to console in order to see if the programs
>>> get
>>>> excluded within killing:
>>>> echo "/usr/comoonics/killall5 -15 ${KILLALL_OPTS}"
>>>   >  just before it is called and you should see if it is called
>>> properly.
>>>
>>> Did that, too, and that looks fine. But killall5 never returns, and
>>> never prints any output.
>  >
>> Never returns is suspicious. Try to call it yourself.
> 
> I did - and it really doesn't return.
> I'm a bit stumped at this, especially since the effect is the same both 
> on the original killall5 that ships with OSR sysvinit and my modified 
> killall5, so I'm reasonably sure that killall5 isn't broken or miscompiled.

And just to confirm, I used the same binary on another machine 
(standalone, no OSR or clustering), and it works exactly as expected 
(prints out what processes it is killing). That means that whatever 
causes killall5 to go away and never return is specific to glfs+OSR 
(since killall5 works fine on my gfs+OSR clusters). I'm not sure where 
to even begin debugging this, though, so any ideas would be welcome.

Gordan

Re: [OSR-devel] killall5 output

[Marc G. <gr...@at...>]

----- "Gordan Bobic" <go...@bo...> wrote:

> Gordan Bobic wrote:
> > On 26/12/2009 21:44, Marc Grimme wrote:
> > 
> And just to confirm, I used the same binary on another machine 
> (standalone, no OSR or clustering), and it works exactly as expected 
> (prints out what processes it is killing). That means that whatever 
> causes killall5 to go away and never return is specific to glfs+OSR 
> (since killall5 works fine on my gfs+OSR clusters). I'm not sure where
> 
> to even begin debugging this, though, so any ideas would be welcome.
You might want to try to start it with strace. I recall something that under some environments the browsing through /proc which is done by killall5 freezes. And I think this is done before killing. Somehow what does not work is a stat call on some /proc files within /proc/<pid>. I don't recall exactly but I have something like this in mind.

If you have found the pid that causes the problem perhaps we get some new ideas on how to handle this behaviour.

Marc.

Re: [OSR-devel] killall5 output

[Gordan B. <go...@bo...>]

Marc Grimme wrote:

>> And just to confirm, I used the same binary on another machine 
>> (standalone, no OSR or clustering), and it works exactly as expected 
>> (prints out what processes it is killing). That means that whatever 
>> causes killall5 to go away and never return is specific to glfs+OSR 
>> (since killall5 works fine on my gfs+OSR clusters). I'm not sure where
>>
>> to even begin debugging this, though, so any ideas would be welcome.
 >
> You might want to try to start it with strace. I recall something that
> under some environments the browsing through /proc which is done by
> killall5 freezes. And I think this is done before killing. Somehow what
> does not work is a stat call on some /proc files within /proc/<pid>. I
> don't recall exactly but I have something like this in mind.
> 
> If you have found the pid that causes the problem perhaps we get some
> new ideas on how to handle this behaviour.

OK, I have straced killall5, and the last few things it does is stat 
/proc/version (twice, it seems) and set up SIGTERM, SIGSTOP and SIGKILL 
signals. This appears to correspond to lines 682-692 in killall5.c:

mount_proc();
...
signal(SIGTERM, SIG_IGN);
signal(SIGSTOP, SIG_IGN);
signal(SIGKILL, SIG_IGN);

The last thing strace reports is:

kill(4294967295,SIGSTOP

(note - no closing bracket)

which seems to correspond to line 695:

if (TEST == 0) kill(-1, SIGSTOP);

Reading what "man 2 kill" says:
POSIX.1-2001 requires that kill(-1,sig) send sig to all processes that 
the current process may send signals to, except possibly for some 
implementation-defined system processes.

I have a suspicion that this may well be the cause of the problems. 
killall5 doesn't iterate through all the processes to kill! According to 
this, sending "kill(-1, <signal>)" sends the signal to all the processes 
that we have permissions to terminate without explicitly specifying the 
processes to terminate! Since killall5 is running as root at this point, 
this means all processes, with the possible exception of "some 
implementation-defined system processes". Right now my bet would be on 
this killing glusterfsd (which is in fact running in userspace, and thus 
is extremely unlikely to be exempt).

This brings up another issue - it sounds like the -x option may be 
ineffective, too, even on the normal GFS related processes. If the 
signals get sent to all processes, then this would include the the 
processes specified by -x, regardless. This leads me to suspect that 
unless these processes are explicitly excluded in the kernel 
implementation, they are not spared the killing at this stage. Looking 
at the ps output - fenced, groupd, aisexec and ccsd, for example, don't 
show up in square brackets, which implies they aren't running in kernel 
space (although that isn't really definitive, only indicative, AFAIK). 
So, this may be affected by the bug, too - but this may not be obvious 
because once they die, the node will get fenced by the other nodes, 
which will end up doing something similar. Or maybe these processes 
simply catch and ignore the signals if they are being used (e.g. if gfs 
is mounted), or something like that. Anyway, that is just hypothesis at 
this point, but it's probably worth checking if you have a suitable test 
environment handy (I don't have a non-production gfs cluster handy at 
the moment).

Anyway, I'm going to comment out line 695 and see how that goes. In 
theory, this seems superfluous anyway, since the iteration through /proc 
for processes to kill should catch everything anyway, and in fact, it is 
this iteration that -x relies on for it's functionality! Otherwise 
kill(-1) will just blow everything away and preempt anything -x might do 
in the first place!

Am I missing something obvious here? Is there a flaw in my analysis?

Gordan

Re: [OSR-devel] killall5 output

[Gordan B. <go...@bo...>]

Gordan Bobic wrote:
> Marc Grimme wrote:
> 
>>> And just to confirm, I used the same binary on another machine 
>>> (standalone, no OSR or clustering), and it works exactly as expected 
>>> (prints out what processes it is killing). That means that whatever 
>>> causes killall5 to go away and never return is specific to glfs+OSR 
>>> (since killall5 works fine on my gfs+OSR clusters). I'm not sure where
>>>
>>> to even begin debugging this, though, so any ideas would be welcome.
>  >
>> You might want to try to start it with strace. I recall something that
>> under some environments the browsing through /proc which is done by
>> killall5 freezes. And I think this is done before killing. Somehow what
>> does not work is a stat call on some /proc files within /proc/<pid>. I
>> don't recall exactly but I have something like this in mind.
>>
>> If you have found the pid that causes the problem perhaps we get some
>> new ideas on how to handle this behaviour.
> 
> OK, I have straced killall5, and the last few things it does is stat 
> /proc/version (twice, it seems) and set up SIGTERM, SIGSTOP and SIGKILL 
> signals. This appears to correspond to lines 682-692 in killall5.c:
> 
> mount_proc();
> ...
> signal(SIGTERM, SIG_IGN);
> signal(SIGSTOP, SIG_IGN);
> signal(SIGKILL, SIG_IGN);
> 
> The last thing strace reports is:
> 
> kill(4294967295,SIGSTOP
> 
> (note - no closing bracket)
> 
> which seems to correspond to line 695:
> 
> if (TEST == 0) kill(-1, SIGSTOP);
> 
> Reading what "man 2 kill" says:
> POSIX.1-2001 requires that kill(-1,sig) send sig to all processes that 
> the current process may send signals to, except possibly for some 
> implementation-defined system processes.
> 
> I have a suspicion that this may well be the cause of the problems. 
> killall5 doesn't iterate through all the processes to kill! According to 
> this, sending "kill(-1, <signal>)" sends the signal to all the processes 
> that we have permissions to terminate without explicitly specifying the 
> processes to terminate! Since killall5 is running as root at this point, 
> this means all processes, with the possible exception of "some 
> implementation-defined system processes". Right now my bet would be on 
> this killing glusterfsd (which is in fact running in userspace, and thus 
> is extremely unlikely to be exempt).
> 
> This brings up another issue - it sounds like the -x option may be 
> ineffective, too, even on the normal GFS related processes. If the 
> signals get sent to all processes, then this would include the the 
> processes specified by -x, regardless. This leads me to suspect that 
> unless these processes are explicitly excluded in the kernel 
> implementation, they are not spared the killing at this stage. Looking 
> at the ps output - fenced, groupd, aisexec and ccsd, for example, don't 
> show up in square brackets, which implies they aren't running in kernel 
> space (although that isn't really definitive, only indicative, AFAIK). 
> So, this may be affected by the bug, too - but this may not be obvious 
> because once they die, the node will get fenced by the other nodes, 
> which will end up doing something similar. Or maybe these processes 
> simply catch and ignore the signals if they are being used (e.g. if gfs 
> is mounted), or something like that. Anyway, that is just hypothesis at 
> this point, but it's probably worth checking if you have a suitable test 
> environment handy (I don't have a non-production gfs cluster handy at 
> the moment).
> 
> Anyway, I'm going to comment out line 695 and see how that goes. In 
> theory, this seems superfluous anyway, since the iteration through /proc 
> for processes to kill should catch everything anyway, and in fact, it is 
> this iteration that -x relies on for it's functionality! Otherwise 
> kill(-1) will just blow everything away and preempt anything -x might do 
> in the first place!
> 
> Am I missing something obvious here? Is there a flaw in my analysis?

Sorry, small ammendment - line 695 only sends SIGSTOP. Since it resumes 
the processes afterwards, this may not affect all processes, e.g. those 
required by gfs. But if it sends a stop to glusterfsd, it's almost 
certain that rootfs will in fact block, so it is definitely an issue for 
that. Since SIGSTOP cannot be caught or ignored by the process itself, 
killall5 will have to be explicitly modified to do this differently, 
e.g. using a double-pass through /proc, specifically without including 
glusterfsd in the list of processes to signal.

Gordan

Re: [OSR-devel] killall5 - proposed patch

[Gordan B. <go...@bo...>]

Gordan Bobic wrote:
> Gordan Bobic wrote:
>> Marc Grimme wrote:
>>
>>>> And just to confirm, I used the same binary on another machine 
>>>> (standalone, no OSR or clustering), and it works exactly as expected 
>>>> (prints out what processes it is killing). That means that whatever 
>>>> causes killall5 to go away and never return is specific to glfs+OSR 
>>>> (since killall5 works fine on my gfs+OSR clusters). I'm not sure where
>>>> to even begin debugging this, though, so any ideas would be welcome.
>>  >
>>> You might want to try to start it with strace. I recall something that
>>> under some environments the browsing through /proc which is done by
>>> killall5 freezes. And I think this is done before killing. Somehow what
>>> does not work is a stat call on some /proc files within /proc/<pid>. I
>>> don't recall exactly but I have something like this in mind.
>>>
>>> If you have found the pid that causes the problem perhaps we get some
>>> new ideas on how to handle this behaviour.
>> OK, I have straced killall5, and the last few things it does is stat 
>> /proc/version (twice, it seems) and set up SIGTERM, SIGSTOP and SIGKILL 
>> signals. This appears to correspond to lines 682-692 in killall5.c:
>>
>> mount_proc();
>> ...
>> signal(SIGTERM, SIG_IGN);
>> signal(SIGSTOP, SIG_IGN);
>> signal(SIGKILL, SIG_IGN);
>>
>> The last thing strace reports is:
>>
>> kill(4294967295,SIGSTOP
>>
>> (note - no closing bracket)
>>
>> which seems to correspond to line 695:
>>
>> if (TEST == 0) kill(-1, SIGSTOP);
>>
>> Reading what "man 2 kill" says:
>> POSIX.1-2001 requires that kill(-1,sig) send sig to all processes that 
>> the current process may send signals to, except possibly for some 
>> implementation-defined system processes.
>>
>> I have a suspicion that this may well be the cause of the problems. 
>> killall5 doesn't iterate through all the processes to kill! According to 
>> this, sending "kill(-1, <signal>)" sends the signal to all the processes 
>> that we have permissions to terminate without explicitly specifying the 
>> processes to terminate! Since killall5 is running as root at this point, 
>> this means all processes, with the possible exception of "some 
>> implementation-defined system processes". Right now my bet would be on 
>> this killing glusterfsd (which is in fact running in userspace, and thus 
>> is extremely unlikely to be exempt).
>>
>> This brings up another issue - it sounds like the -x option may be 
>> ineffective, too, even on the normal GFS related processes. If the 
>> signals get sent to all processes, then this would include the the 
>> processes specified by -x, regardless. This leads me to suspect that 
>> unless these processes are explicitly excluded in the kernel 
>> implementation, they are not spared the killing at this stage. Looking 
>> at the ps output - fenced, groupd, aisexec and ccsd, for example, don't 
>> show up in square brackets, which implies they aren't running in kernel 
>> space (although that isn't really definitive, only indicative, AFAIK). 
>> So, this may be affected by the bug, too - but this may not be obvious 
>> because once they die, the node will get fenced by the other nodes, 
>> which will end up doing something similar. Or maybe these processes 
>> simply catch and ignore the signals if they are being used (e.g. if gfs 
>> is mounted), or something like that. Anyway, that is just hypothesis at 
>> this point, but it's probably worth checking if you have a suitable test 
>> environment handy (I don't have a non-production gfs cluster handy at 
>> the moment).
>>
>> Anyway, I'm going to comment out line 695 and see how that goes. In 
>> theory, this seems superfluous anyway, since the iteration through /proc 
>> for processes to kill should catch everything anyway, and in fact, it is 
>> this iteration that -x relies on for it's functionality! Otherwise 
>> kill(-1) will just blow everything away and preempt anything -x might do 
>> in the first place!
>>
>> Am I missing something obvious here? Is there a flaw in my analysis?
> 
> Sorry, small ammendment - line 695 only sends SIGSTOP. Since it resumes 
> the processes afterwards, this may not affect all processes, e.g. those 
> required by gfs. But if it sends a stop to glusterfsd, it's almost 
> certain that rootfs will in fact block, so it is definitely an issue for 
> that. Since SIGSTOP cannot be caught or ignored by the process itself, 
> killall5 will have to be explicitly modified to do this differently, 
> e.g. using a double-pass through /proc, specifically without including 
> glusterfsd in the list of processes to signal.

Attached is a proposed patch that tries to work around this specific 
issue. It seems to work the machine no longer locks up on killall5, 
which is a good sign, and a definitive improvement. :^)

Please review.

Now the problem is that md devices get stopped shortly afterwards, just 
after the "INIT: no more processes left in this run-level" message. Now 
I have to figure out what does that, since these must remain running 
until the shutdown sequence reaches the OSR initroot... But that's 
something for a separate thread.

Gordan