#124 Asynchronous OWLMaker

[Vidar Hasfjord]

I need to learn modern asynchronous programming, and in that regard, I thought it would be a good idea to use OWLMaker as a study case, as there are some asynchronous features I would like to see implemented. With that in mind, I have now created a new feature branch, "tools/OWLMaker/branches/asynch", for experimental work on an asynchronous version of OWLMaker that can:

• launch multiple builds in parallel,
• does not freeze during the build process,
• shows progress in the status bar (while the program window continues normal operation),
• can cancel the build cleanly on user request [feature-requests:#118].

Feel free to help out.

[Ognyan Chernokozhev]

Great idea! Can't wait to see it.

[Vidar Hasfjord]

After some fairly simple changes, based on std::async, I now have a version that can launch builds in parallel. There is some work to do to handle build message output (now jumbled), and "vc.mak" does not cope well with concurrency. But, MSBuild for the Clang-based Embarcadero compilers seems to handle it (mostly), which dramatically shortens build time for large batches for these compilers, as they do not support multiprocessing within each build. I can build 16 libraries for BCC64 in 4 minutes now, compared to 30+ minutes before. See [r3982].

[Vidar Hasfjord]

I have now implemented asynchronous handling of CM_OWLMAKER, so that the program no longer freezes during the build process. See [r3986].

The only thing I had to do was to split CmOWLMaker into two, with CmOWLMaker opening the wizard and then, if the wizard is not cancelled, launching the build asynchronously (using std::async). The rest of the function has been refactored into CmShowBuildResult, which is called at the end of the build task by posting WM_COMMAND with CM_SHOWBUILDRESULT. I also added a command enabler to disable CM_OWLMAKER while a build is in progress, hence avoiding concurrent invocations.

To facilitate these changes, TOWLMakerClient::OpenWizard had to be split into three; with OpenWizard now just doing that; opening the wizard and returning the dialog return value. The rest of the function has been refactored into Build and ShowBuildResults. Build is executed asynchronously by CmOWLMaker, while ShowBuildResult is called by CmShowBuildResult.

Curiously, these changes make the program do a clean cancel of the build, if you select the parallel build option. You can then close the console, and the console will apparently send a Ctrl+C signal, and the main window will just report build failure. However, if the parallel build option is not selected, the whole program closes.

[Vidar Hasfjord]

The asynchronous handling of CM_OWLMAKER using std::asynch seems to work very well. There is more to do, but to simplify further development, I now intend to merge my changes back to the mainline. If you have any objections, let me know.

[Ognyan Chernokozhev]

Great work, Vidar!
I have no objections, only perhaps before that to build and upload a new version of OWLMaker for download, so that contains all the other fixes.

Jogy

[Vidar Hasfjord]

Ok. I will create a tag before integrating the asynch changes into the mainline.

[Vidar Hasfjord]

I have now committed numerous changes (see log) which implement

• build progress indication,
• optional console and
• build cancel function.

Remaining issue: Aborting a build make subsequent builds fail until the program is restarted.

Edit: This issue has now been fixed in [r4196]. The trick was to pass the build command to "CMD.EXE /C", rather than execute it directly. It seems this isolates OWLMaker from any corruption of the program environment caused by an interrupted build.

"vc.mak" does not cope well with concurrency.

Well, it does now! Revision [r4198] makes a change to the VC++ makefiles to support parallel builds. It has been merged into 6.44 in [r4199], and it seems to work well. Here are some numbers from my testing:

OWLMaker, Visual Studio 2017 toolset, AMD Ryzen 7 1700 (16-thread) CPU,
8 GB RAM, Samsung SSD 850 Pro: 

Rebuild OWLNext 7 (trunk), all 32 configurations:
  multi-threaded (/MP), parallel: 3:13
  multi-threaded (/MP), serial: 5:27
  single-threaded, serial: 14:59

Rebuild OCFNext/OWLExt/CoolPrj 7 (trunk), all 80 configurations, 
  multi-threaded (/MP), parallel: 1:49
  multi-threaded (/MP), serial: 5.28
  single-threaded, serial: 8:47

Note that CL.EXE may occasionally produce "Internal Compiler Error" with full parallel build. Out of memory? Internal race condition (e.g. temporary file clash)?

Parallel builds with Embarcadero Clang-based toolsets may also fail occasionally , due to a race condition on the "owl.res" file produced in the project directory. I presume this could be circumvented by telling the compiler to use the intermediate directory instead. However, my RAD Studio trial has run out, so I don't know how to modify the project files (help if you can).

For BCC32, the parallel build option has been disabled altogether (in [r4201]). The toolset seems to use internal temporary files that cause race condition (clash) between parallel builds.

In other news, OWLMaker now disables the Close button in the console window, and sets the Ctrl handler to ignore Ctrl+C input [r4204]. OWLMaker now also ensures that dependent libraries are built in the right order (setup before core before extension libraries) [r4203].

TODO:

• Implement nicer handling of the output from parallel builds. See [feature-requests:#65].

I have uploaded a Debug build to Files/tools. Try it out!

[Vidar Hasfjord]

Erwin Lotter wrote:

Btw.: Is std::async as you used it for OWLMaker (really nice results!) a kind of replacement for OMP?

In a sense. Both allow you to exploit parallelism. However, std::async is simply an abstraction on top of std::thread that makes it easier to launch an asynchronous task and get the result back to the caller (thread). Here is a couple of discussions about std::async vs OpenMP:

https://stackoverflow.com/questions/47895048/c-async-vs-openmp-tasks
https://stackoverflow.com/questions/23258037/openmp-vs-c11-threads

As I understand it, OpenMP can be more suited for massive parallelism, although std::asynch is more useful for asynchronous programming (e.g. responsive UI).

A problem with the implementation of std::asynch is that it will mindlessly create threads beyond the capability of the hardware (instead of using a thread pool calibrated to the hardware resources available). Too many threads create overhead. For example, if you launch a full rebuild of all the configurations of OWLNext 6.44 with the Visual Studio toolset (that's 112 configurations), you will push Windows and your machine pretty hard. Have a look in Task Manager while it runs, and you will see an awful lot of threads created and memory used.

[Vidar Hasfjord]

I have now released Build 4204. Hopefully it works well.