Re: [edk2-buildtools] [edk2] hang/deadlock with MultiThread build in build.py due to use of output pipes in popen() call

[Andrew F. <af...@ap...>]

One of my coworkers point out this: http://www.dabeaz.com/python/GIL.pdf about how python runs threads very slow, and slows down on multicore systems. 

So I guess I have to ask the question of why we have to wrap building the code via nmake or make in python? This seems like it could really slow down the builds. 

This brings up a 2nd question. I tried playing around with the makefiles that are autogenrated. I noticed I could do a clean, but the rebuild fails. Why is that? Is the top level makefile not used in the build process? 

Would it be possible to do the build step from python and without python and see what the speed difference is? I don't think nmake is parallel, but make supports parallel build. So it would be better to test with make vs. python. 

If python really is slowing down the build, we could have C code to invoke the python and then invoke the make commands in some form of parallelism. 

Andrew Fish





On Mar 19, 2010, at 12:29 PM, Jim Buteau wrote:

> Hello,
> 
> My development team has been plagued by edk2 builds sporadically hanging. Our build environment is cygwin (1.5 and 1.7).  Our python version is 2.5.2.  When the hang occurs the windows task manager shows a make.exe process apparently stuck as indicated by its cpu usage.  After much experimentation I have learned/concluded the following
> 
> - The hung task was spawned in LaunchCommand(Command, WorkingDir) with the following python line:
>    Proc = Popen(Command, stdout=PIPE, stderr=PIPE, env=os.environ, cwd=WorkingDir, bufsize=-1)
> 
> - The subsequent call to Proc.wait() never returns.
> 
> - The use of Proc.wait() is somewhat troubling given the following (from http://docs.python.org/library/subprocess.html)    
> WarningThis will deadlock if the child process generates enough output to a stdout or stderr pipe such that it blocks waiting for the OS pipe buffer to accept more data. Use communicate() to avoid that.
> 
> - Attempts to replace the wait() call with poll() or communicate() did not work for me.
> 
> - Removing the I/O redirection appears to (after hundreds of multi-threaded builds) solve the hang problem.  The popen() call was changed to 
>    Proc = Popen(Command, env=os.environ, cwd=WorkingDir, bufsize=-1)
> The threads created to read the Proc stdout and stderr pipes were removed.  In this solution stdout and stderr from make go directly to the console instead of passing through EdkLogger.  Given the alleviation of our hang problem, this is acceptable to us.
> 
> Is this problem known by others?  Can it be solved without removing the output redirection?
> 
> Thanks,
> Jim Buteau
> 
> 
> ------------------------------------------------------------------------------
> Download Intel® Parallel Studio Eval
> Try the new software tools for yourself. Speed compiling, find bugs
> proactively, and fine-tune applications for parallel performance.
> See why Intel Parallel Studio got high marks during beta.
> http://p.sf.net/sfu/intel-sw-dev_______________________________________________
> edk2-devel mailing list
> edk...@li...
> https://lists.sourceforge.net/lists/listinfo/edk2-devel