[edk2-buildtools] hang/deadlock with MultiThread build in build.py due to use of output pipes in popen() call

[Jim B. <bu...@ap...>]

[sorry - this was posted to edk2-devel before finding this list]

Hello,

My development team has been plagued by edk2 builds sporadically hanging. Our build environment is cygwin (1.5 and 1.7).  Our python version is 2.5.2.  When the hang occurs the windows task manager shows a make.exe process apparently stuck as indicated by its cpu usage.  After much experimentation I have learned/concluded the following

- The hung task was spawned in LaunchCommand(Command, WorkingDir) with the following python line:
   Proc = Popen(Command, stdout=PIPE, stderr=PIPE, env=os.environ, cwd=WorkingDir, bufsize=-1)

- The subsequent call to Proc.wait() never returns.

- The use of Proc.wait() is somewhat troubling given the following (from http://docs.python.org/library/subprocess.html)    
WarningThis will deadlock if the child process generates enough output to a stdout or stderr pipe such that it blocks waiting for the OS pipe buffer to accept more data. Use communicate() to avoid that.

- Attempts to replace the wait() call with poll() or communicate() did not work for me.

- Removing the I/O redirection appears to (after hundreds of multi-threaded builds) solve the hang problem.  The popen() call was changed to 
   Proc = Popen(Command, env=os.environ, cwd=WorkingDir, bufsize=-1)
The threads created to read the Proc stdout and stderr pipes were removed.  In this solution stdout and stderr from make go directly to the console instead of passing through EdkLogger.  Given the alleviation of our hang problem, this is acceptable to us.

Is this problem known by others?  Can it be solved without removing the output redirection?

Thanks,
Jim Buteau