#1579 Build system enhancements

[T-640]

Before beginning I'd like to give a bit of context. I have used Qt Creator and the QBS build system before. It has some similarities to your build system:

1) Their projects are roughly equivalent to your workspace files;
2) Their products are more or less like your project files;
3) They provide a scripting language, ECMAScript in their case.

Their syntax is a QML dialect, which itself has many similarities with JSON. And while you use XML, the ideas behind using these formats for project description are more or less the same.

So coming from QBS, adjusting to your build system was not hard. I did not even use GUI to configure projects and jumped straight into the file formats description and scripting commands.

Below is the list of features that are present in QBS that I would like to be added to the build system of Code::Blocks. I have managed to implement some of these features myself via a Squirrel script, you can find it in the attachment below. The attachment also includes a sample project built with the help of this Squirrel script, and .qbs files to show comparison between QBS and your build system. Note that the Squirrel script I wrote is incomplete and might break in some cases because I focused only on static libraries. I am using it on a Windows 10 64-bit machine with w64devkit.

Without further ado, here is what I would like to see in the build system of Code::Blocks:

1. More automation, less manual work

In dependent project (say, executable depending on a static library) the user has to manually specify the following:

1) Paths to files whose changes must trigger relinking, static libraries in particular <Option external_deps=""/>. Othes were discussing that this is inconvenient here;
2) The include directories of dependencies <Add directory=""/>)
3) The name of every single library file produced by dependencies, for linking <Add library=""/>;
4) The directories for the linker to search for libraries <Add directory=""/>

It is very inconvenient, making the whole build process more fragile as the project grows. With so much to be done manually, changing names, moving files and directories as the project evolves becomes a daunting task. Dependencies should automatically fill the depending projects with that information, dependent projects should not need to be aware of internal layout of their dependencies.

QBS automatically takes care of all that. So does my Squirrel script in the attachment. Nonetheless I think this script is not a solution but rather a workaround for the time being. This functionality should probably be implemented within the code of the build system itself.

5) Installation of files. In QBS project's output like executables, DLLs and any other files can be marked for installation. QBS creates a special install-root directory where it copies them. This allows to automatically make a complete package once the compilation is done, which is ready to be deployed anywhere.

6) As far as I understand, configurations for different operating systems, CPU architectures and compilers require separate cbp files, or at least a completely separate section in that file. Having quickly peeked at the Code::Blocks's source code I see that there are two files, CodeBlocks_wx32-unix.cbp and CodeBlocks_wx32.cbp. They share a long list of same files to be compiled. That is a lot of repetition! That means that every time you move, remove or change file's name you have to edit both .cbp files. Which is very inconvenient. In contrast in QBS for a multi-platform project I have a single file, with a single set of C++ files listed. For many common compiler options, like setting the C++ standard version that build system provides an abstraction, so I just set it to whatever version I need without having to memorize the particular flags of various compilers. And if there are some specific compiler options not covered by that abstractions I can conditionally include them. Same goes for CPU architectures or operating systems, I can conditionally include extra .cpp files depending on what I am compiling the project for.

2. The location of various files produced by projects

1) The default directory for the output of the build process (object files, static and dynamic library files, executable files, etc.) should be outside of the directory where the source code is. I believe that the source code directory should be left untouched.

In the script attached I set it to $WORKSPACE_DIR/$WORKSPACE_NAME-build/. In that directory every project creates its own subdirectory where the object files and the rest of the output is stored.
Compile the attached project to see what I mean. Ideally I'd like this build folder containing all the output to be one step above above the workspace directory in the filesystem hierarchy. This is how it's done in QBS.

2) Speaking of the source code directory being polluted with unrelated files, Code::Blocks creates .layout and .depends files there. Unfortunately I could not find a way to set a different location for them via script. Could you make it so they are put in that external build output directory I mentioned above as well? I believe it would be elegant to store all the output, including these files in an external directory. And I am not the only one who would like to see it that way.

3. Changes to .cbp and .workspace files

1) Move the project dependency declaration to the .cbp file. The projects themselves should know what they depend on, instead of being supplied with such vital information from an external file. With the current setup I will have to declare project dependencies all over again if I start working on a new piece of software and reuse code from the previous one. If projects themselves could declare their dependencies just copying the needed projects and setting paths in the workspace file would be enough. This is how it is done in QBS. Their equivalent of the workspace file provides only paths to their equivalent of the project files. Their equivalent of project files declares dependency via the project's title, and the equivalent of the workspace file performs the search.

2) Allow workspaces to include other workspaces. Having a single workspace for dozens of projects is inconvenient. Suppose there is a program that has both client and server infrastructure. Ideally I would have the Main.workspace which includes Client.workspace and Server.workspace both residing in their respective directories. And instead of being all intermingled in GUI, all projects related to client and server parts will be displayed in their allotted tree view. When working on the client part the server part could be collapsed, hiding the projects that are not being worked on at the moment. This is done in QBS, their "workspaces" can include other "workspaces". To see how it looks in GUI view the image attached below

3) Make project titles optional. If cbp file is missing the title attribute then the cbp file name without the extension should become the title.

4. Related to build scripts

1) There is a very annoying bug making debugging build scripts difficult.

2) Allow build scripts to be run during and after the build process. As far as I understand currently they can be executed before the build process only. This could be useful for the ttCommandsOnly TargetType.

3) Let the .cbp file to embed the script code, just like HTML files can embed ECMAScript code as an alternative to using separate files for scripting. This would be very useful for small scripts that don't really need a whole separate file dedicated to them.

4) In the build script of a random project the $(PROJECTNAME) expands not to the name of that project, but to the name of the current active project. In my build script I wanted to use it to setup per-project build directories, like this $WORKSPACE_DIR/$WORKSPACE_NAME-build/$PROJECTNAME/ but had to use cbase.GetTitle() instead.

5) Not sure if really a problem or my misunderstanding of some of the API bindings, but calling CompileTargetBase::GetTitle (and I suppose any other function returning wxString) does not work well when it comes to setting a slot of a Squirrel table:

function SetBuildOptions (cbase)
{
    local table = {}
    local title = cbase.GetTitle()

    table [title] <- 1              // OK
    table ["title"] <- 1            // OK
    table [title.tostring()] <- 1   // OK
    table [cbase.GetTitle()] <- 1   // OK

    table [title]                   // OK
    table ["title"]                 // OK
    table [title.tostring()]        // Ok
    table [cbase.GetTitle()]        // the index 'instance' does not exist
}

5. Related to build scripts documentation

1) In documentation clarify what cbase actually represents in the mandatory SetBuildOptions function. It says here that "The "cbase" parameter in this function is a CompiletargetBase instance." Does it represent a project? Or a build target when the script is run at the build target level? Something else? It would also be useful to add a typeof metamethod for every custom class, so calling "typeof cbase" returns an actual class name instead of just "instance".

2) Clarify documentation about the GetVar and SetVar functions. They seem to function differently depending on the class derived from CompileOptionsBase. I have accidentally found out that they work on cbProject, calling SetVar and then ProjectManager::SaveProject saves variable in the <Environment> block of the .cbp file. This is not listed in documentation here. In all other cases the GetVar and SetVar functions appear to work only within the context of the build script they were called from. Initially I thought to use them to exchange exported data between projects (like include directories, etc.) but later decided to simply use global tables to store such data with project's titles acting as slots for these tables.

3) ProjectManager::AskForBuildTargetIndex is missing despite being listed here

4) While the names of various scripting commands listed here are self-explanatory, pulling documentation from the header files instead of just pointing them out for further reading would be helpful.

6. Project file syntax

The .cbp file syntax is simple. I had no problems understanding it. However, from what I can tell it was not really meant to be read and written by a human being, but rather for the GUI to take advantage of the underlying XML parsing library. So the expected workflow of Code::Blocks is to examine and write settings to the project files via GUI as far as I can tell.

But I find it redundant. I never needed GUI to do that, editing the .cbp file via text editor was simple enough. Moreover the project file with a simple syntax like that acts as a piece of documentation. When I am working with QBS files taking a quick glimpse on the project file that I haven't been working on in a while and thus forgot some things helps setting me back on track.

Which brings me to the main point: perhaps it would be better to have a JSON-like syntax (similar to QBS) for the project files instead of the current XML one? The former is easier to read, there is less repetition, and writing comments in XML is rather inconvenient. Here is how a .cbp project file can be rewritten in such hypothetical syntax:

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<CodeBlocks_project_file>
    <FileVersion major="1" minor="6" />
    <Project>
        <Option title="Program" />
        <Option compiler="gcc" />
        <Build>
            <Script file="Program.script" />
            <Target title="Release">
                <Option type="1" />
                <Option compiler="gcc" />
                <Option external_deps="$WORKSPACE_DIR/$WORKSPACE_NAME-build/Library/Library.lib;$WORKSPACE_DIR/$WORKSPACE_NAME-build/DeeperLibrary/DeeperLibrary.lib" />
            </Target>
            <Linker>
                <Add library="Library" />
                <Add library="DeeperLibrary" />
                <Add directory="$WORKSPACE_DIR/$WORKSPACE_NAME-build/Library" />
                <Add directory="$WORKSPACE_DIR/$WORKSPACE_NAME-build/DeeperLibrary" />
            </Linker>
        </Build>
        <Compiler>
            <Add directory="../Library" />
            <Add directory="../DeepLibrary" />
        </Compiler>
        <Unit filename="Main.cpp"/>
        <Extensions />
    </Project>
</CodeBlocks_project_file>

Project
{
    title: "Program"
    compiler: "gcc"

    Build
    {
        scripts: ["Program.script"]
        Target
        {
            type: executable
            compiler: "gcc"
            external_deps:
            [
                "$WORKSPACE_DIR/$WORKSPACE_NAME-build/Library/Library.lib",
                "$WORKSPACE_DIR/$WORKSPACE_NAME-build/DeeperLibrary/DeeperLibrary.lib"
            ]
        }
        Linker
        {
            libraries: ["Library", "DeeperLibrary"]
            directories:
            [
                "$WORKSPACE_DIR/$WORKSPACE_NAME-build/Library/",
                "$WORKSPACE_DIR/$WORKSPACE_NAME-build/DeeperLibrary/"
            ]
        }
    }

    Compiler
    {
        include_directories: ["../Library", "../DeeperLibrary"]
    }

    files: "Main.cpp"
}

Conclusion

To sum up, trying the build system of Code::Blocks after working with QBS was quite a significant step back. The script that I wrote resolves the major issues, but there is still a lot of work that needs to be done, as you can see. Unless there are some critical bugs to be fixed, the build system should likely have the top priority, as it is the only one that Code::Blocks can work with.

If that is too much, could you perhaps adopt the QBS build system as a plugin? Be it your own build system being enhanced or some other one like QBS getting support (but not CMake please!), either way works for me. There is also another build system called Xmake which looks like it has similar ideas to QBS and is based on Lua, never used it personally though. Here are the basic examples of it

[T-640]

Before trying the .qbs files in the attachment please delete the "Workspace.qbs.user" file (just the ".user" file, not the "Workspace.qbs" itself). I forgot to do it and apparently it is impossible to reupload the attachment or edit the initial message altogether. This file is something that Qt Creator makes, in some way similar to .layout and .depends files that Code::Blocks writes.