#70 Support for the Clang-based Embarcadero compilers

[Vidar Hasfjord]

Support for building OWLNext with the Clang-based compilers in Embarcadero C++Builder/RAD Studio is only partly implemented. Full support should be added and backported to supported release branches.

[Vidar Hasfjord]

As I know Codeguard it's not compatible with CLang.

Good, one configuration less to worry about. :-)

No problem in support single/multi/VCL threaded.

And what about dynamic linking? I guess some users will want to build a DLL version of the library.

Without dynamic linking mode, we are now at 24 configurations. With dynamic linking that doubles to 48 configurations. Hopefully, you will not run into a RAD Studio limitation. :-)

Tip: Start with a lower number of configurations until you have solved all the issues mentioned and have a fully working project file for those configurations.

[Vidar Hasfjord]

I have now added support for Precompiled Headers (PCH) for "source/owlcore/CB" on the trunk [r3950]. Support in 6.44 for "source/owlcore/CBXE3" was merged in [r3953].

With PCH, build times within IDE and OWLMaker are now halved.

Here are build times with and without PCH on my system (Ryzen 7 1700, 8GB DDR4-2400):

OWLNext 7, RAD Studio 10.2 Tokyo, build times within IDE:

  "Debug Win64"
  Elapsed time: 00:03:53.8 - <owl/pch.h> (_OWLPCH and _OWLALLPCH not defined)
  Elapsed time: 00:03:49.7 - empty "pch.h"
  Elapsed time: 00:03:38.4 - no PCH (turned off in options)
  Elapsed time: 00:02:59.7 - <owl/pch.h>, _OWLALLPCH
  Elapsed time: 00:02:17.0 - <owl/defs.h>
  Elapsed time: 00:02:13.4 - <owl/defs.h> and select standard headers
  Elapsed time: 00:01:45.1 - <owl/pch.h>, _OWLPCH

  "Debug Win32"
  Elapsed time: 00:01:48.8 - <owl/pch.h>, _OWLPCH

OWLNext 7 and 6.44, RAD Studio 10.2 Tokyo, build times within OWLMaker:

  owl-7.0.0-c730-x86-dt.lib Success 0:04:42 - no PCH
  owl-7.0.0-c730-x86-dt.lib Success 0:02:18 - _OWLPCH
  owl-7.0.0-c730-x64-dt.lib Success 0:02:15 - _OWLPCH

  owl-6.44.1-c730-x86-dt.lib Success 0:03:04 - _OWLPCH
  owl-6.44.1-c730-x64-dt.lib Success 0:02:16 - _OWLPCH

For comparison, OWLNext 7, Visual Studio 2017, build times within OWLMaker:

  owl-7.0.0-v1910-x86-dt.lib Success 0:00:09 - _OWLPCH
  owl-7.0.0-v1910-x64-dt.lib Success 0:00:09 - _OWLPCH

[Vidar Hasfjord]

Hi Sebastian,

In my work on the new asynchronous OWLMaker, I have found that builds with Embarcadero Clang-based toolsets may fail occasionally , due to a race condition on the "owl.res" file produced in the project directory. I presume this could be circumvented by telling the compiler to put this file in the intermediate directory instead, i.e. the same directory where object files are created.

However, my RAD Studio trial has run out, so I don't know how to modifify the project files. Can you help?

[Vidar Hasfjord]

I am surprised you C++Builder users didn't know this (or if you did, why you didn't tell me): Parallel compilation for C++Builder project files can be enabled quite simply for the Clang-based compilers. See Embarcadero documentation | Clang-enhanced C++ Compilers.

Note that parallel compilation is not compatible with precompiled headers. Still, enabling parallel compilation on 4 cores or more is a clear win. On my 16-thread AMD Ryzen 7 1700 CPU and 8 GB DDR4 2400 MT/s RAM, I get these numbers:

OWLNext Debug x86 configuration:

  PCH, serial compilation: 2:27
  No PCH, parallel compilation (j=16): 0:36

OWLNext Debug, Release, x86 and x64 (4 configurations, parallel build):

  PCH, serial compilation: 2:40
  No PCH, parallel compilation (j=16): 2:14
  No PCH, undefined _OWLPCH, parallel compilation (j=16): 2:08

[Vidar Hasfjord]

I have now added the possibility to use build multiprocessing with the Embarcadero Clang-based toolset for OWLNext 7 (trunk). Support was enabled in the project files in [r4316] and in OWLMaker in [r4317].

If "Build with multiple processes" is selected, OWLMaker will pass "SubProcessesNumber=%NUMBER_OF_PROCESSORS%" to MSBuild. For example:

rem Build owl-7.0.0-c730-x86-t.lib

"C:\Program Files (x86)\Embarcadero\Studio\19.0\bin\rsvars.bat" &
msbuild CB\owl.cbproj /t:Clean;Build /p:Config=Release;
  Platform=Win32;
  OWLNextVersion=7.0.0;
  OWLNextCompilerVersion=730;
  OWLNextSetup=false;
  SubProcessesNumber=%NUMBER_OF_PROCESSORS%

Note that this will use all threads available, e.g. 16 threads (logical processors) on a 8-core AMD Ryzen 7 1700 CPU with SMT. This will be taxing on the system, so it can be wise to set "Process priority" to "Low" as well to improve system responsiveness during the build. Combining "Build with multiple processes" and "Build configurations in parallel" is especially taxing, and will eat a lot of memory.

Also note that the command "Stop the build" does not work very well with multiprocessing and the Clang-based toolsets. The stop command is not heeded until all builds in progress are completed. TODO: Make OWLMaker properly abort the child processes.

Finally, note that parallel builds may sometimes fail due to data race (resource contention) on some files. Relaunching the build with the option "Delete old output files before build" deselected (i.e. so only the unsuccessful files are rebuilt) usually succeeds.

[Ognyan Chernokozhev]

There is support for the latest Cland Embarcadero compilers both in the 6.44 branch and in the trunk. This ticket can be closed.

[Vidar Hasfjord]

Hi Jogy, the Clang support, in 6.44 in particular, is only partial. In 6.44, neither OWL5_COMPAT, Unicode, nor dynamic linking are supported, and multi-threaded RTL is the only threading mode supported. For version 7/trunk, I have expanded the support to include OWL5_COMPAT and Unicode, but that's it.

There is also the issues mentioned in earlier comment posts here, with using custom build parameters in the MSBuild project files. This is brittle and does not survive after the user saves the project from the IDE. As discussed, all build configurations should be set up in the IDE, so that OWLMaker could simply select one when invoking the build (rather than passing custom build parameters).

[Sebastian Ledesma]

Update / Restart:

I've used OWL6.44 current branch (6.44.11) and made a copy of owl.cbproj and ocf.cbproj, for testing purposes of Win64+unicode.

So I:
1 - Enabled UNICODE: In both projects I've set the option _TCHAR maps as: wchar_t ( internally in the .cbproj the node is called _TCHARMapping )

2 - Make a small change in the post-build events to ensure that the generated lib will have the correct name and so to avoid mistakes.
Ie:
copy $(OUTPUTPATH) ......\lib\ocf-6.44.11-c720-x64-tu.lib

Both projects builded without any problem. So at least there arent any errors when building.

Then I've downaloaded the FontText example, selected Win64, and enabled UNICODE ( _TCHAR maps to wchar_t) . I've builded the proyect with just a minimal update in the code, and after that it started seamlessly.

I've checked in the taskManager that the process doesnt have the "32 bits" attribute and that it also showed correctly the arabic and hebrew charactes.

Sebas

[Vidar Hasfjord]

Hi Sebastian, good work on testing Unicode compatibility with 6.44 and the Clang toolset.

Note that the Clang toolset supports both Unicode and OWL5_COMPAT mode in version 7/trunk (neither is supported for 6.44). If you want to bring support for these build modes to 6.44, you can probably just merge and adapt these changes (see code logs). OWLMaker will also need updating, of course.

Then I've downaloaded the FontText example,

By the way, in version 7/trunk, this example is part of "examples/classes" now, as "Examples | TStatic" (static.cpp), demonstrating the new SetWindowFont and SetTextColor features in version 7, but still retaining the 6.44 compatible code, conditionally compiled based on OWLVersion. It also automatically adds the Arabic and other exotic character set strings for the "Hello world" greeting, if compiled in Unicode mode. So you can merge this to the 6.44 branch as well, if you want.

PS. Did you know that since Windows Version 1903 (May 2019 Update), Windows now supports the UTF-8 code page in narrow character set (ANSI) mode? It would be cool to test that with OWLNext, and if that works well, then we should say something about it in the Knowledge Base and FAQ. UTF-8 for international text processing has become the de-facto standard for the web and all major OS platforms now (with Windows also adding support).

https://docs.microsoft.com/en-us/windows/uwp/design/globalizing/use-utf8-code-page

[Sebastian Ledesma]

1 - I've repeated the test with the libraries owlext and coolprj: created a copy of the originals .cbproj file, set the global option _TCHAR maps to: wchar_t, adjust the post build commands and builded the projects- Both compiled and builded without problem. I didnt tested them with am application.

2 - In the other hamd I've played a little with the command line options of MSBUILD, I was looking how to set the _TCHARMapping property. So I added /p:_TCHARMapping=wchar_t and builded the library.
Ie:
"C:\Program Files (x86)\Embarcadero\Studio\18.0\bin\rsvars.bat" & (if exist ....\lib\owl-6.44.11-c720-x64-tu.lib del ....\lib\owl-6.44.11-c720-x64-tu.lib) & msbuild CBXE3\owl.cbproj /t:Clean;Build /p:Config=Release;Platform=Win64 /p:_TCHARMapping=wchar_t && ren ....\lib\owl-6.40-b14-x64-t.lib owl-6.44.11-c720-x64-tu.lib

Some warning showed, but the library was built. After that I've tested with the OWL FontText example and it worked.

3 - i've did a simple test of OWLnext FontText builded as ANSI and adding the manifest. It didnt worked. I could made a mistake but I'm pretty sure whats wrong and at the end of the day there is no magic bullet. I will investigate a little more and create a post on the open discussion forum.

[Vidar Hasfjord]

1

Great! Good work on the testing!

2

That's very interesting. Since the property _TCHARMapping has no Condition attribute specified within the project file, I didn't think you could override it from the command line.

In version 7/trunk, I've created OWLNext specific properties that override the internal project properties conditionally. Perhaps that was not necessary after all, and we could just override the real project properties by passing them on the command line directly. That would mean less sensitivity to changes to the project file also, which would be very nice.

That said, there is the issue about the name of the target files. For 6.44, OWLMaker needs to rename the libraries correctly after the build, while for 7/trunk this is handled in the project file as a PostBuildEvent. Perhaps we simply can override the target name by passing some property on the command line? Is there a TargetName property? Or perhaps the target name is simply based on SanitizedProjectName and we can pass that?

All that said, as mentioned earlier in this discussion, the best solution would be to have all the supported configurations set up within the project file with the correct target names, and then just pass the selected Platform and Config parameters. Then the project file would be equally useful for OWLMaker and the user opening it in the IDE. Setting up all the supported configurations shouldn't be that much work, since you can use inheritance internally in the project files. Most of the settings would be in the "Base" PropertyGroup, and then the PropertyGroup for each configuration would have just a few overriding properties.

3

You mean you tested setting the default code page to UTF-8 in ANSI mode, but it didn't work? Not even passing a UTF-8 string to MessageBoxA? You might want to confirm that the code page is really set to UTF-8 by making a system call to retrieve the current code page.

Anyway, interesting stuff. I'm currently translating my OWLNext application. With Windows now supporting UTF-8, I think I may rule out moving to UTF-16. It all depends on whether UTF-8 is workable with resource files (which probably must be stored in UTF-16; I doubt the resource compiler supports UTF-8).

But this off-topic. Like you say, we should take the UTF-8 discussion elsewhere.

[Sebastian Ledesma]

I did a little more research.

/p:BCC_UseClassicCompiler=true / false
allows to use the BCC32 (Classic Borland Compiler) or the BCC32c (Clang).
Please note that:
a - The classic compiler it's only available in Win32. In Win64 and others this property it's ignored.
b - The project already includes the property BCC_UseClassicCompiler= false, so when building for Win32 using MSBUILD with owl.cbproj we already are using Bcc32c and there is no need to override this property in the command line.

I've did the confirmation by simple renaming the bcc32 & bcc32c and setting the property true and false just to make sure that the build fails when the requested compiler it's missing.

So the next question was if it's possible to define single/multithread at command line.
/p:Multithreaded=true / false
should do that.
The command:

"C:\Program Files (x86)\Embarcadero\Studio\18.0\bin\rsvars.bat" & (if exist ..\..\lib\owl-6.44.11-c720-x64-s.lib del ..\..\lib\owl-6.44.11-c720-x64-s.lib) & msbuild CBXE3\owl.cbproj /t:Clean;Build /p:Config=Release;Platform=Win64 /p:Multithreaded=false && ren ..\..\lib\owl-6.40-c720-x64-t.lib owl-6.44.11-c720-x64-s.lib

builds the library, , but I still have to find a way to verify that is a single thead library.

Also:
/p:AppType=Library
and
/p:ProjectType=CppDynamicLibrary
should allow to build a .dll, but some it's missing as in the final process I'm getting this error:

"C:\OWL644\source\owlcore\CBXE3\owl.cbproj" (Clean;Build target) (1) ->
(_PerformBCCILink target) ->
C:\Program Files (x86)\Embarcadero\Studio\18.0\Bin\CodeGear.Cpp.Targets(3592,5): error : Fatal: Unable to open file '
OWL-6.44.11-C720-X64-T.LIB'

Sebas

Edited to correct the example of building single/multihread by command line.

[Sebastian Ledesma]

Related to the last item, it seems that my build process it's trying to link against the OWL-6.44.11-C720-X64-T.LIB instead of building that (or OWL.dll).
Surely I'm missing some define like _BUILDOWLDLL and _OWLDLL

[Sebastian Ledesma]

Regarding to:

So the next question was if it's possible to define single/multithread at command line.
/p:Multithreaded=true / false
should do that.
The command:

"C:\Program Files (x86)\Embarcadero\Studio\18.0\bin\rsvars.bat" & (if exist ....\lib\owl-6.44.11-c720-x64-s.lib del ....\lib\owl-6.44.11-c720-x64-s.lib) & msbuild CBXE3\owl.cbproj /t:Clean;Build /p:Config=Release;Platform=Win64 /p:Multithreaded=false && ren ....\lib\owl-6.40-c720-x64-t.lib owl-6.44.11-c720-x64-s.lib
builds the library, , but I still have to find a way to verify that is a single thead library.

According to http://docwiki.embarcadero.com/RADStudio/Sydney/en/Predefined_Macros
the MT=1 is be defined when selecting Multithreaded=true, so we have a way to detect if the project is being built with/without multithread in the source code.

Sebas

[Vidar Hasfjord]

So the next question was if it's possible to define single/multithread at command line.

I don't see the need for the single-threaded option. What's lacking in OWLMaker for OWLNext 7, when selecting an Embarcadero toolset, is the dynamic linking build mode (and perhaps the VCL-compatible threading mode, if there is still a need for it). With that in place, the build options available for Embarcadero toolsets will match the build options available for VC++.

[Anonymous]

I agree with you, I personally need the 64 bit DLL and maybe (not sure) the VCL mode. Not the single threded. I have tried to modify the cbproj, by changing the StatiLibrary flag into CppDynamicLibrary, and it seems to compile but not to link, as you also reported. I also added the BUILDOWLDLL and OWLDLL compiler options (with the underscore) but I have an internal compiler error (I'm using the 10.4.2 Community edition) if I enable the PCH, and a linker error if I disable the precompiled header:

[ilink64 Error] Error: Unresolved external 'OwlMain(int, char*)' referenced from C:\OWL644\SOURCE\OWLCORE\CBXE3\WIN64\RELEASE\MAIN.O
[ilink64 Error] Error: Public symbol 'owl::InitGlobalModule(HINSTANCE__)' defined in both module C:\OWL644\SOURCE\OWLCORE\CBXE3\WIN64\RELEASE\GLOBAL.O and C:\OWL644\SOURCE\OWLCORE\CBXE3\WIN64\RELEASE\MODULE.O
[ilink64 Error] Error: Public symbol 'owl::GetGlobalModule()' defined in both module C:\OWL644\SOURCE\OWLCORE\CBXE3\WIN64\RELEASE\GLOBAL.O and C:\OWL644\SOURCE\OWLCORE\CBXE3\WIN64\RELEASE\MODULE.O
[ilink64 Error] Error: Public symbol 'owl::Module' defined in both module C:\OWL644\SOURCE\OWLCORE\CBXE3\WIN64\RELEASE\GLOBAL.O and C:\OWL644\SOURCE\OWLCORE\CBXE3\WIN64\RELEASE\MODULE.O

Any suggestions?

Thanks, Luigi

[Luigi Bianchi]

Just an update: I've been able to compile the DLL, but I still don't know if it works!!! I will update you and if someone is interested in the updated cbproj file, please let me know.

Luigi

[Vidar Hasfjord]

Luigi wrote:

Just an update: I've been able to compile the DLL, but I still don't know if it works!!!

Sounds promising — good work!

Personally I have no experience with the Embarcadero tools, but hopefully Sebastian, Jogy or someone else still using them will join in and help you complete this work. If not, feel free to take ownership of this ticket, and if you need to, try to recruit help in the discussion forum. When you have something ready I will help test and release it.

I've added your profile (@luigibianchi) to the developer list, though you may already have been on the list with another profile (@luigi65), unless that is a different person.

[Vidar Hasfjord]

Luigi (@luigibianchi) committed changes to support VCL build mode on the 7.0 release branch in [r6263]-[r6265] (tab regression was fixed in [r6267]). See log. These changes were merged into the trunk in [r6271].