Re: [edk2-buildtools] [edk2] hang/deadlock with MultiThread build in build.py due to use of output pipes in popen() call

[Andrew F. <af...@ap...>]

Ken,

We have been seeing hangs is OS X, Cygwin, and Windows.   We are seeing  hangs with MAX_CONCURRENT_THREAD_NUMBER = 1.

make has a -j (--jobs=) option that allows parallel make assuming the makefiles are constructed properly. 

So maybe the simple solution would look like:
build --preprocess
make --jobs --directory=$(WORKSPACE)\Build\NT32\DEBUG_VS2003\IA32
build --postprocess

Have the 1st phase of build process the DSC, DEC, INF files and autogen all the code. Then use make to do a parallel build. Then call build to post process the results of the build, like parse the FDF file and build the FVs.

If the makefiles generated by build worked we could test the performance difference. 

I also think calling make directly would cause a lot of our hangs to go away. We could keep the current build scheme for nmake (it is really optimized for that) and add the ability to do a preprocess and postprocess step into  the build command for people who wanted to use make. 


In a lot of our projects we find we don't end up using the target.txt file and we have a script or makefile to build our project. Given this it would be easy to call build, then make, then build.  For example on the beableboard we have to generate a binary file that is used by the MASK ROM to turn on memory prior to buiding and after the build we generate debugger scripts in the projects Build directory. 

Andrew Fish





On Mar 19, 2010, at 7:46 PM, Lu, Ken wrote:

> Hi, Andrew:
>          I have a question; you mentioned here is multi-process or multi-thread? Do you mean when you enable MUTIL_THREAD in Conf/target.txt or from build command?
>  
>          The article you provide is a good investigation, and need a time to research….but I also find similar performance issue in wxPython environment. So personally, I almost give up use wxPython to write GUI but use C#/mono.
>         
>          I like you idea about use C code for muti-threading building, but it make big different between OS, as your known, threading.Thread in python is really a very basic wrapper. Because threading implementation is very dependent OS’s API and usage. I think Our build tool need try *not* use OS dependent code, …

The tools are all coded to POSIX, so I think it would be possible. But using make would be simpler. 

>          I guess the threading schedule cost which bring from muti-threading is much less than performance tuning which bring from multi-threading. So seems it is not much valuable for use C code to wrapper threading. Do you agree?

After thinking about it I think using make is better. 

>  
>          I notice you find a new issue, I want to confirm, do you mean after all make files are created, then you go to the folder of top makefile and type “nmake clean”, it does not work?

If I run build for my project. Then I cd into Build\NT32\DEBUG_VS2003\IA32. nmake clean. nmake. The nmake will fail. 

The error message is can not open file  ...object_flles.lst. 


>          Thanks.
>  
> Best Regard
> K
> ---------------------------
> hear and you forget; see and you remember; do and you understand
>  
> From: Andrew Fish [mailto:af...@ap...] 
> Sent: 2010年3月20日 6:17
> To: edk...@li...
> Subject: Re: [edk2-buildtools] [edk2] hang/deadlock with MultiThread build in build.py due to use of output pipes in popen() call
>  
> One of my coworkers point out this: http://www.dabeaz.com/python/GIL.pdf about how python runs threads very slow, and slows down on multicore systems. 
>  
> So I guess I have to ask the question of why we have to wrap building the code via nmake or make in python? This seems like it could really slow down the builds. 
>  
> This brings up a 2nd question. I tried playing around with the makefiles that are autogenrated. I noticed I could do a clean, but the rebuild fails. Why is that? Is the top level makefile not used in the build process? 
>  
> Would it be possible to do the build step from python and without python and see what the speed difference is? I don't think nmake is parallel, but make supports parallel build. So it would be better to test with make vs. python. 
>  
> If python really is slowing down the build, we could have C code to invoke the python and then invoke the make commands in some form of parallelism. 
>  
> Andrew Fish
>  
>  
> 
> 
>  
> On Mar 19, 2010, at 12:29 PM, Jim Buteau wrote:
> 
> 
> Hello,
>  
> My development team has been plagued by edk2 builds sporadically hanging. Our build environment is cygwin (1.5 and 1.7).  Our python version is 2.5.2.  When the hang occurs the windows task manager shows a make.exe process apparently stuck as indicated by its cpu usage.  After much experimentation I have learned/concluded the following
>  
> - The hung task was spawned in LaunchCommand(Command, WorkingDir) with the following python line:
>    Proc = Popen(Command, stdout=PIPE, stderr=PIPE, env=os.environ, cwd=WorkingDir, bufsize=-1)
>  
> - The subsequent call to Proc.wait() never returns.
>  
> - The use of Proc.wait() is somewhat troubling given the following (from http://docs.python.org/library/subprocess.html)    
> Warning
> This will deadlock if the child process generates enough output to a stdout or stderr pipe such that it blocks waiting for the OS pipe buffer to accept more data. Use communicate() to avoid that.
> 
>  
> - Attempts to replace the wait() call with poll() or communicate() did not work for me.
>  
> - Removing the I/O redirection appears to (after hundreds of multi-threaded builds) solve the hang problem.  The popen() call was changed to 
>    Proc = Popen(Command, env=os.environ, cwd=WorkingDir, bufsize=-1)
> The threads created to read the Proc stdout and stderr pipes were removed.  In this solution stdout and stderr from make go directly to the console instead of passing through EdkLogger.  Given the alleviation of our hang problem, this is acceptable to us.
>  
> Is this problem known by others?  Can it be solved without removing the output redirection?
>  
> Thanks,
> Jim Buteau
>  
>  
> ------------------------------------------------------------------------------
> Download Intel® Parallel Studio Eval
> Try the new software tools for yourself. Speed compiling, find bugs
> proactively, and fine-tune applications for parallel performance.
> See why Intel Parallel Studio got high marks during beta.
> http://p.sf.net/sfu/intel-sw-dev_______________________________________________
> edk2-devel mailing list
> edk...@li...
> https://lists.sourceforge.net/lists/listinfo/edk2-devel
>