Menu

#11 Hang on zombie child process

Unstable (example)
closed-fixed
nobody
None
5
2022-07-05
2020-02-07
No

The original code is doing:
1. fork child process (ssh)
2. sigprocmask to catch SIGCHLD
3. wait on signal (or terminal data) on pselect

In case ssh exits before 2. the SIGCHLD is lost and process hangs on
pselect. This was proven on large ansible playbook running on dozens of
remote hosts with ssh control persist feature enabled. In such
configuration password propmt is almost never presented and if the
operation on remote is really simple and host is under heavy load (ie.
when executing concurrent ansible tasks in dozens forks), the issue is
more likely to manifest. In other cases the race condition is too tiny.
In case ssh password prompt is sent to the client by remote, the issue
does not manifest, as the child will always wait until terminal action
occurs.

I attached a patch that fixes this issue. It was confirmed on a large ansible playbook that hangs with the original code.

1 Attachments

Discussion

  • Shachar Shemesh

    Shachar Shemesh - 2020-02-07

    Hi Grzegorz,

    Thank you for the report and the patch. Upon reviewing the code, I can confirm the problem as reported. I do not, however, agree with the approach you took to fix it. I prefer that the loop not be restructured.

    I am attaching my own proposal to fixing this issue, by moving the sigprocmask to before the fork. Please see if it if you have any comments before I commit it.

    Thanks again,
    Shachar

     
  • Grzegorz Sikorski

    I did not want to do this, as it is affecting forked process sigmask, which may have other unexpected consequences. Are you sure it is safe to change default sigmask for ssh?

     

    Last edit: Grzegorz Sikorski 2020-02-07
  • Grzegorz Sikorski

    OK, sorry, I see you are reverting the mask to original in child. This looks fine to me.

     
    • Shachar Shemesh

      Shachar Shemesh - 2020-02-07

      Can you test it in your environment to make sure it indeed solves the problem?

       
  • Shachar Shemesh

    Shachar Shemesh - 2020-02-07

    Read my patch again. It resets the sigprocmask for the forked process before actually starting it (line 303 of the file after applying the patch).

    Without this, it would, indeed, break stuff. As is, it should be perfectly fine.

     
    • Grzegorz Sikorski

      As mentioned, it is ok. Let me test this.

       
    • Grzegorz Sikorski

      Hi Shachar,

      I have tested your fix and it is working. However after deeper review of your patch I think it affects child process somehow, as you are defaulting its mask to block all. This may not be an intention of the sshpass caller, so I propose attached solution instead. Attached patch was also successfully tested.

       
      • Shachar Shemesh

        Shachar Shemesh - 2020-02-07

        Au contraire. It sets the mask for the new process to all signals unblocked. It is using signmask_select, which is initialized with sigemptyset.

        Since I'm starting a brand new session here, I prefer to set the mask to a known state, rather than inherit whatever crap I-don't-know-who sent my way when running sshpass.

        If you say you've tested it and it works, I'm committing my version.

         
        • Grzegorz Sikorski

          Correct, I misread the code. This does not really make much difference as subprocess should take care of signal mask itself and not rely upon what is send to it by parent. The only possible issue I can see here is you are changing the behaviour of the utility, which may affect some users.

           
  • Shachar Shemesh

    Shachar Shemesh - 2020-02-07
    • status: open --> closed-fixed
     
  • Przemyslaw Strzelczak

    Hi Shachar,
    could you release a version with this patch? It is likely that I hit issue described here in my work.

     
  • Przemyslaw Strzelczak

    Noticed it is already released in 1.07.

     

Log in to post a comment.

Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.