Recent changes to BuildAutomation

BuildAutomation modified by Joseph Koshy

Joseph Koshy — Fri, 05 Feb 2021 18:44:36 -0000

--- v20
+++ v21
@@ -2,16 +2,14 @@

 This wiki page describes a proposed build system for the [Elftoolchain project](Home).

-This build system is intended to be "*P2P Transparent*" (the precise definition of this term is given further down this article).
+This build system is intended to be *"P2P Transparent"* (the precise definition of this term is given further down).

 ## Goals

-This build system allows developers to **share their build cycles**, while also being assured of the **integrity of their builds**.
-
 The major goals for the system are:

--   To implement repeatable & hermetic (i.e., fully deterministic) builds from source code.
--   To support "build transparency": to offer the ability to verify that an output artifact (say the ISO image for an installer) had been verifiably built from a specific set of human-reviewed source files.
+-   To support repeatable & hermetic (i.e., fully deterministic) builds from source code.
+-   To support "build transparency": the ability to verify that an output artifact (say the ISO image for an installer) had been verifiably built from a specific set of human-reviewed source files.
 -   To support cross-building of source code.
 -   To allow developers to securely share build cycles with each other (peer-to-peer sharing of their build effort).
 -   To be frugal in its machine resource usage.
@@ -19,11 +17,21 @@

 ## Terminology
 ### Repeatable Builds
-#### Local Repeatability
-
-For the purposes of this document, a build is said to *locally repeatable* when two attempts at building the same source tree on a given machine would result in the 'same' bits being generated.  (I'm using 'same' here as shorthand for "logically indistinguishable by a subsequent build step").
-
-Many compilation processes do not guarantee local repeatability out of the box.  Local non-deterrminism may be caused by the presence of timestamps in the output of a build step, embedded temporary filenames in generated output, output differences caused by unstable orderings of inputs, the use of special macros like `__DATE__` and `__TIME__`, and the like.
+
+**Local Repeatability**
+
+We consider a build to  be locally repeatable when two attempts at building the same source tree on a given machine result in the 'same' effective bits being generated.
+
+The word 'same' is used here to mean "logically indistinguishable when used by a subsequent build step". 
+
+Many compilation processes do not guarantee local repeatability out of the box.   Some causes of local non-repeatibility include:
+
+*   The presence of compilation timestamps in the output of a build step,
+*   Embedded names of temporary files in generated output,
+*   Differences caused by unstable orderings of inputs,
+*   The use of special macros like `__DATE__` and `__TIME__`,
+
+and the like.

 Prior work in making builds locally repeatable includes:

@@ -32,34 +40,38 @@
 -  [Reproducible Builds in NetBSD](https://blog.netbsd.org/tnf/entry/netbsd_fully_reproducible_builds), from the [NetBSD](https://www.netbsd.org/) project.
 -  [Deterministic Builds](https://blog.torproject.org/deterministic-builds-part-one-cyberwar-and-global-compromise) at the [Tor Project](https://www.torproject.org/).

-Some Elftoolchain tools support locally repeatable output (usually specified using the `-D` option).
-
-A locally repeatable build allows the build process to be treated as a *pure function* mapping a set of source artifacts to a set of built artifacts.  Local repeatability would be the first step towards sharing build cycles across multiple developers.
-
-#### Cross-Machine Repeatability a.k.a. Global Reproducibility
-
-In a globally reproducible build, builds on different machines generate the 'same' (i.e., logically indistinguishable) output.  Two different machines are permitted to have different build execution graphs, as long as the final outputs are the 'same'.  For example, the build on one machine could have additional instrumentation interleaved between its build steps, or could use tools that differ from the other.
-
-Some sources of "non-local" non-determinism include: the use of embedded (absolute) pathnames in generated output, build time dependencies on the time zone of the build machine, and so on.
-
-Global reproducibility is needed before we can meaningfully share build cycles across machines.
+Some Elftoolchain tools produce locally repeatable output (this mode is usually specified using the `-D` option).
+
+A locally repeatable build permits the build process to be treated as a *pure function* mapping a set of sources to a set of built artifacts.  
+
+Local repeatability is the first step towards sharing build cycles across multiple developers.
+
+**Cross-Machine Repeatability** a.k.a. **Global Reproducibility**
+
+In a globally reproducible build, builds on different machines generate the 'same' (i.e., logically indistinguishable) output.  Two different machines are permitted to have different build execution graphs, as long as the final outputs are the 'same'.  For example, the build on one machine could have additional instrumentation interleaved between its build steps, or could use tools that differ from the other, provided the end result is the 'same'.
+
+Some sources of "non-local" non-determinism include:
+
+*   The use of embedded (absolute) pathnames in generated output,
+*   Build time dependencies on the time zone of the build machine,
+
+and so on.
+
+Global reproducibility is needed before we can share build cycles across machines.

 ### Hermetic Builds

-In a *hermetic build* every input to the build process is fully specified.  These inputs include:
+In a hermetic build all inputs to the build process is fully specified.  These include:

 - The version of the compilation tools in use.
 - The configuration used when invoking compilation tools: invocation flags, linker scripts, and so on.
-- Each and every module that a given step in the build depends on.  This includes the headers, libraries and configuration traditionally considered to be part of the "base system".
-- And of course the notional "source code" being built.
+- Each and every input that a given step in the build depends on.  This includes the headers, libraries and configuration traditionally considered to be part of the base system, apart from the source code being used as input.

 A hermetic build process guarantees that the build does not depend upon inputs that are "external" in some fashion (and are thus uncontrolled).

-One way to achieve a hermetic build would be have the build framework invoke the individual steps in a build in an controlled environment that contains precisely those files that were explicitly determined to be inputs for the build step (e.g. the build step would be run in a [chroot](https://en.wikipedia.org/wiki/Chroot), a [jail](https://www.freebsd.org/doc/handbook/jails.html) or a symlink tree to a set of read-only inputs).
-
-If build configuration is tracked in version control alongside the rest of the source tree, and if the source code to the compilation tools is also part of the source tree (e.g. as in [NetBSD](https://www.netbsd.org/docs/guide/en/part-compile.html)) then the build can be made *hermetic through time*.  With a hermetic-through-time build we would be able to roll back a source tree to a prior revision and be able to build exactly the same output using the exact same tooling as would have been originally built at that revision.
-
-###  Peer To Peer Builds
+If the source code to the compilation tools is part of the source tree being built (e.g. as is true for [NetBSD](https://www.netbsd.org/docs/guide/en/part-compile.html)) then the build can be made *hermetic through time*.  With a hermetic-through-time build we would be able to roll back a source tree to a prior revision and be able to reconstruct binaries using the exact toolchain that had been originally used at that revision.
+
+**Peer To Peer Builds**

 Once builds are *hermetic* and *globally reproducible*, we would be able to share builds across build peers.

@@ -71,39 +83,39 @@

 Related work: [Bazel][bazel] supports caching of build results using a (centralized) [remote cache](https://docs.bazel.build/versions/master/remote-caching.html).

-#### Herd Immunity and Decentralized Verification
-
-Peer-to-peer builds could also be used to implement a form of "herd immunity".  Once we have multiple builders building the same source tree and sharing metadata about their builds, it would become possible to detect output artifacts that had been tampered with.
+**Herd Immunity** and **Decentralized Verification**
+
+Peer-to-peer builds could also be used to implement a form of herd immunity.  Once we have multiple builders building the same source tree and sharing metadata about their builds, it would become possible to detect output artifacts that had been tampered with.

 This tamper detection protocol could be used to verify, in a decentralized fashion, that the final artifact generated by a build (say the ISO image containing an OS installer) had not been tampered with locally.

-### Auditable Builds
-
-This kind of build process generates a (cryptographically signed) trail of the inputs, outputs and tools used for each build step.
+**Auditable Builds**
+
+This kind of build generates a (cryptographically signed) trail of the inputs, outputs and tools used for each build step.

 This trail could be used to verify the build 'end to end'.

-### P2P Auditable Builds
+**P2P Auditable Builds**

 In this document this term means an extension of auditable builds that share outputs and build trails in a P2P fashion.

-### Vetted Source
+**Vetted Source**

 Source code that is linked to a 'is-reviewed' bit issued by a human.

-### Vetted Builds
+**Vetted Builds**

 These are builds whose source code inputs can be proven to have been vetted by one or more humans.

-### Transparent Builds
-
-I am using this term to mean an *auditable* and *vetted* build.
-
-### P2P Transparent Build
+**Transparent Builds**
+
+This is a build that is both *auditable* and *vetted*.
+
+**P2P Transparent Build**

 This is a *transparent* build which uses a P2P build-sharing backend.

-### Cross-Organisational Verifiability
+**Cross-Organisational Verifiability**

 Sometimes a software build would take for its input one or more binary blobs built by external organisations, for which source code is not available.

@@ -113,7 +125,7 @@

 TODO(jkoshy): Look for existing standards and protocols implementing cross-organisational verifiability, if these exist.

-### The Build Configuration Language
+## The Build Configuration Language

 BSD operating systems are built from source using [BSD make][make] (e.g., the [FreeBSD makeworld](https://www.freebsd.org/doc/handbook/makeworld.html) build process, or by using an additional tool that drives [make][], such as in [NetBSD's build.sh](https://www.netbsd.org/docs/guide/en/part-compile.html)).

@@ -123,7 +135,7 @@

 At this point of time I am inclined to use a notation that is close to S-expressions, for the reasons outlined in some of the sections below.

-#### Tooling friendliness of the notation
+### Tooling friendliness of the notation

 The build configuration language should be tooling-friendly. Large projects almost always need such tooling support. This means that the build configuration language should be easy to parse and modify programmatically.

@@ -156,7 +168,7 @@
   deps: (...))
 ```

-#### Support for modularity
+### Support for modularity

 The build system should support the structuring of project code into logical modules.

@@ -164,19 +176,19 @@

 [Make][], in contrast uses a single global namespace, with its attendant lack of modularity.  Large projects then end up using recursive invocations of `make`, which all the drawbacks documented in  [Recursive make Considered Harmful](http://aegis.sourceforge.net/auug97.pdf) (Peter Miller, 2008).

-#### Separation of concerns
+### Separation of concerns

 We would like the configuration used to specify build & test scenarios (i.e., different compilation modes: `-c optimize` vs `-c debug` or running tests under `valgrind` vs running them normally) to be separate from the configuration that specifies the dependencies between input and output artifacts.  These two aspects of build configuration are intertwined in [make][].

 We want users to be able to easily define their own scenarios for building and testing code.

-#### Multiple logically connected build outputs per directory
+### Multiple logically connected build outputs per directory

 The BSD `/usr/share/mk/*.mk` framework currently works best when a single sub-directory in a source tree builds one program or one library. This restriction can be fairly limiting.

 We would like our build system to allow test suites to be co-located with code that they test.   The build system should allow multiple logically related libraries and binaries to be built in a single source directory.

-#### Expressivity
+### Expressivity

 Builds for large projects would need a language for writing "build macros" and the like.

@@ -186,35 +198,35 @@

 If we go the S-expression route, then a declarative subset of R*n*RS Scheme may be a natural choice for the extension language.  

-### Other Features Desired For The Build System
-
-#### Offline Operation
+## Other Features Desired For The Build System
+
+### Offline Operation

 The build system should be able to work in 'offline' mode, without needing external network access to function (although builds would be entirely local in this case).

 This feature would be useful for developers in areas of poor network connectivity.

-#### Frugality
+### Frugality

 The build system should be able to work effectively in a resource constrained host environment (such as a laptop computer).

 [Bazel][] in particular seems to need a lot of memory when building projects.

-#### Cross-platform Builds
+### Cross-platform Builds

 The build system should support building code on a variety of (non-native) machine architectures and operating systems.

 Cross building an operating system often involves compiling the cross compiler itself.  The build tool should be able to handle these kinds of scenarios efficiently.

-#### Building With Local Changes
+### Building With Local Changes

 The build system should be able to work efficiently on a source tree with a small number of local changes, sharing the build of the unmodified parts of the source tree with its peers.

-#### Easing (Build) Portability
+### Easing (Build) Portability

 New target architectures should be easy to support, either through cross compilation, or by invoking the necessary build steps "remotely" using a small, portable helper that runs on the target hardware.

-#### Support for Bootstrapping
+### Support for Bootstrapping

 The build tool should be usable when bootstrapping an operating system from source.

BuildAutomation modified by Joseph Koshy

Joseph Koshy — Mon, 13 Apr 2020 22:11:59 -0000

--- v19
+++ v20
@@ -2,21 +2,28 @@

 This wiki page describes a proposed build system for the [Elftoolchain project](Home).

+This build system is intended to be "*P2P Transparent*" (the precise definition of this term is given further down this article).
+
 ## Goals

-The goals of this system are:
-
--   To implement repeatable & hermetic (i.e., fully deterministic) and verifiable builds from source.
--   To allow developers to securely share build cycles with each other (peer-to-peer sharing of their build).
--   To offer a more expressive build language than [make][make] that is also tooling friendly.
+This build system allows developers to **share their build cycles**, while also being assured of the **integrity of their builds**.
+
+The major goals for the system are:
+
+-   To implement repeatable & hermetic (i.e., fully deterministic) builds from source code.
+-   To support "build transparency": to offer the ability to verify that an output artifact (say the ISO image for an installer) had been verifiably built from a specific set of human-reviewed source files.
 -   To support cross-building of source code.
--   To support "build transparency": the ability to verify that a given binary had been verifiably built from a specific set of inputs.
+-   To allow developers to securely share build cycles with each other (peer-to-peer sharing of their build effort).
 -   To be frugal in its machine resource usage.
-
+-   To offer a more expressive build language than [make][make], that is also tooling friendly.
+
+## Terminology
 ### Repeatable Builds
-**Local Repeatability**
-
-For the purposes of this document, a build is said to *locally repeatable* when two attempts at building the same source tree would result in exactly the same bits being generated.  Many compilation processes do not guarantee local repeatability out of the box.  Local non-deterrminism may be caused by the presence of timestamps in the output of a build step, embedded temporary filenames in generated output, output differences caused by unstable orderings of inputs, the use of special macros like `__DATE__` and `__TIME__`, and the like.
+#### Local Repeatability
+
+For the purposes of this document, a build is said to *locally repeatable* when two attempts at building the same source tree on a given machine would result in the 'same' bits being generated.  (I'm using 'same' here as shorthand for "logically indistinguishable by a subsequent build step").
+
+Many compilation processes do not guarantee local repeatability out of the box.  Local non-deterrminism may be caused by the presence of timestamps in the output of a build step, embedded temporary filenames in generated output, output differences caused by unstable orderings of inputs, the use of special macros like `__DATE__` and `__TIME__`, and the like.

 Prior work in making builds locally repeatable includes:

@@ -29,15 +36,17 @@

 A locally repeatable build allows the build process to be treated as a *pure function* mapping a set of source artifacts to a set of built artifacts.  Local repeatability would be the first step towards sharing build cycles across multiple developers.

-**Cross-Machine Repeatability**
-
-Local repeatability is not sufficient - we need the same output to be generated from a given source tree when built on multiple machines.  Such "non-local" non-determinism would be caused by embedded (absolute) pathnames in generated output, dependencies on the time zone of the build machine, and so on.
-
-For the purposes of this document a build is said to be *cross-machine repeatable* if two builds on two different machines result in identical outputs.
+#### Cross-Machine Repeatability a.k.a. Global Reproducibility
+
+In a globally reproducible build, builds on different machines generate the 'same' (i.e., logically indistinguishable) output.  Two different machines are permitted to have different build execution graphs, as long as the final outputs are the 'same'.  For example, the build on one machine could have additional instrumentation interleaved between its build steps, or could use tools that differ from the other.
+
+Some sources of "non-local" non-determinism include: the use of embedded (absolute) pathnames in generated output, build time dependencies on the time zone of the build machine, and so on.
+
+Global reproducibility is needed before we can meaningfully share build cycles across machines.

 ### Hermetic Builds

-In a *hermetic build* every input to the build process would be fully specified.  These inputs include:
+In a *hermetic build* every input to the build process is fully specified.  These inputs include:

 - The version of the compilation tools in use.
 - The configuration used when invoking compilation tools: invocation flags, linker scripts, and so on.
@@ -46,15 +55,13 @@

 A hermetic build process guarantees that the build does not depend upon inputs that are "external" in some fashion (and are thus uncontrolled).

-One way to achieve a hermetic build would be have the build framework invoke the individual steps in a build in an controlled environment that contains precisely those files that were explicitly determined to be inputs for the build step (e.g., the build step would be run in a [chroot](https://en.wikipedia.org/wiki/Chroot), a [jail](https://www.freebsd.org/doc/handbook/jails.html) or a symlink tree to a set of read-only inputs).
-
-If build configuration is tracked in version control alongside the rest of the source tree, and if the source code to the compilation tools is also part of the source tree (e.g., as in [NetBSD](https://www.netbsd.org/docs/guide/en/part-compile.html)) then the build can be made *hermetic through time*.  With a hermetic-through-time build we would be able to roll back a source tree to a prior revision and be able to build exactly the same output using the exact same tooling as would have been originally built at that revision.
-
-In this document, a build that is both cross-machine repeatable and hermetic is called a *fully deterministic* build.
-
-### Shared Peer To Peer Builds
-
-Once builds are fully deterministic we would be able to share builds across build peers.
+One way to achieve a hermetic build would be have the build framework invoke the individual steps in a build in an controlled environment that contains precisely those files that were explicitly determined to be inputs for the build step (e.g. the build step would be run in a [chroot](https://en.wikipedia.org/wiki/Chroot), a [jail](https://www.freebsd.org/doc/handbook/jails.html) or a symlink tree to a set of read-only inputs).
+
+If build configuration is tracked in version control alongside the rest of the source tree, and if the source code to the compilation tools is also part of the source tree (e.g. as in [NetBSD](https://www.netbsd.org/docs/guide/en/part-compile.html)) then the build can be made *hermetic through time*.  With a hermetic-through-time build we would be able to roll back a source tree to a prior revision and be able to build exactly the same output using the exact same tooling as would have been originally built at that revision.
+
+###  Peer To Peer Builds
+
+Once builds are *hermetic* and *globally reproducible*, we would be able to share builds across build peers.

 ![Peer to peer builds](https://elftoolchain.sourceforge.io/images/p2p-builds.jpg)

@@ -64,14 +71,38 @@

 Related work: [Bazel][bazel] supports caching of build results using a (centralized) [remote cache](https://docs.bazel.build/versions/master/remote-caching.html).

-### Herd Immunity And Peer Verifiability
+#### Herd Immunity and Decentralized Verification

 Peer-to-peer builds could also be used to implement a form of "herd immunity".  Once we have multiple builders building the same source tree and sharing metadata about their builds, it would become possible to detect output artifacts that had been tampered with.

-This tamper detection protocol could be used to verify, in a decentralized fashion, that the final artifact generated by a build (say the ISO image containing an OS installer) had actually been deterministically built from a specific revision of the OSes source code.
-
-For the purposes of this document, a build system that can offer an auditable trail of its build steps is said to be*auditable*. An auditable build process that is coupled with peer-to-peer verification of its build steps is termed a *peer verifiable* build process. 
-
+This tamper detection protocol could be used to verify, in a decentralized fashion, that the final artifact generated by a build (say the ISO image containing an OS installer) had not been tampered with locally.
+
+### Auditable Builds
+
+This kind of build process generates a (cryptographically signed) trail of the inputs, outputs and tools used for each build step.
+
+This trail could be used to verify the build 'end to end'.
+
+### P2P Auditable Builds
+
+In this document this term means an extension of auditable builds that share outputs and build trails in a P2P fashion.
+
+### Vetted Source
+
+Source code that is linked to a 'is-reviewed' bit issued by a human.
+
+### Vetted Builds
+
+These are builds whose source code inputs can be proven to have been vetted by one or more humans.
+
+### Transparent Builds
+
+I am using this term to mean an *auditable* and *vetted* build.
+
+### P2P Transparent Build
+
+This is a *transparent* build which uses a P2P build-sharing backend.
+  
 ### Cross-Organisational Verifiability

 Sometimes a software build would take for its input one or more binary blobs built by external organisations, for which source code is not available.
@@ -96,23 +127,28 @@

 The build configuration language should be tooling-friendly. Large projects almost always need such tooling support. This means that the build configuration language should be easy to parse and modify programmatically.

-[Bazel][] and [Buck][] use a syntax reminiscent of Python for some aspects of their build rules although other aspects, such as build dependencies, are specified using a different (embedded) syntax. Build extensions for these systems are written using a Python-like extension environment; please see [Starlark](https://docs.bazel.build/versions/master/skylark/language.html) for [Bazel][], and [Buck Macros](https://buck.build/extending/macros.html) for [Buck][].
+Due to historical reasons [Bazel][] and [Buck][] use a syntax reminiscent of Python for their build rules. Other aspects, such as the naming of build dependencies, use a different (embedded) syntax.  Extensions to the build system's rules are written using a Python-like extension environment; please see [Starlark](https://docs.bazel.build/versions/master/skylark/language.html) for [Bazel][], and [Buck Macros](https://buck.build/extending/macros.html) for [Buck][].

 ```python
-  # Defines the build process for library "libfoo".
-  cc_library(
+# A Bazel/Buck example.
+#
+# The following rule invocation specifies the build process for library "libfoo".
+cc_library(
     name = "foo",
     srcs = ["foo.c", "bar.c"],
     hdrs = ["foo.h"],
     deps = ["//some/package"],  # dependencies use a different sub-language
-  )
+)
 ```

-We could adopt this syntax (for continuity), or we could alternately use a notation that is based on S-expressions which may be easier to extend.  For example, using an [SRFI-88/89](https://srfi.schemers.org/srfi-88/srfi-88.html)-esque syntax:
+Although we could adopt this syntax (for continuity), I have a slight preference for a more uniform notation, say one that is based on S-expressions.
+
+For example, using an [SRFI-88/89](https://srfi.schemers.org/srfi-88/srfi-88.html)-esque syntax:

 ```scheme
- ; Defines the build process for library "libfoo".
- ; proposal only - this may not be the final syntax.
+ ; This stanza might define the build process for library "libfoo".
+ ;
+ ; This is a proposal only - this may not be the final syntax.
  (c_library
   name: "foo"
   srcs: ("foo.c" "bar.c")

BuildAutomation modified by Joseph Koshy

Joseph Koshy — Tue, 18 Feb 2020 16:07:57 -0000

--- v18
+++ v19
@@ -10,6 +10,7 @@
 -   To allow developers to securely share build cycles with each other (peer-to-peer sharing of their build).
 -   To offer a more expressive build language than [make][make] that is also tooling friendly.
 -   To support cross-building of source code.
+-   To support "build transparency": the ability to verify that a given binary had been verifiably built from a specific set of inputs.
 -   To be frugal in its machine resource usage.

 ### Repeatable Builds

BuildAutomation modified by Joseph Koshy

Joseph Koshy — Sat, 04 Jan 2020 16:31:27 -0000

--- v17
+++ v18
@@ -21,6 +21,7 @@

 -  [Reproducible Builds](https://reproducible-builds.org/docs/) by the [Debian](https://www.debian.org/) project.
 -  [Reproducible Builds in FreeBSD](https://people.freebsd.org/~emaste/2017-03-12-AsiaBSDCon-Reproducible-Builds-FreeBSD.pdf) from the [FreeBSD](https://www.freebsd.org/) project.
+-  [Reproducible Builds in NetBSD](https://blog.netbsd.org/tnf/entry/netbsd_fully_reproducible_builds), from the [NetBSD](https://www.netbsd.org/) project.
 -  [Deterministic Builds](https://blog.torproject.org/deterministic-builds-part-one-cyberwar-and-global-compromise) at the [Tor Project](https://www.torproject.org/).

 Some Elftoolchain tools support locally repeatable output (usually specified using the `-D` option).

BuildAutomation modified by Joseph Koshy

Joseph Koshy — Sun, 29 Dec 2019 20:49:39 -0000

--- v16
+++ v17
@@ -1,6 +1,6 @@
 # Build Automation

-This wiki page describes a proposed build system for the [Elftoolchain project](Home), briefly touching upon some aspects of its design.
+This wiki page describes a proposed build system for the [Elftoolchain project](Home).

 ## Goals

BuildAutomation modified by Joseph Koshy

Joseph Koshy — Sun, 29 Dec 2019 19:58:10 -0000

--- v15
+++ v16
@@ -218,9 +218,9 @@

 Android's build system [Soong][soong] uses a configuration language that is closely related to that used by Bazel.

-[Shake][shake]'s build configuration language is [Haskell](https://www.haskell.org/), in its full generality.
-
-[Dune][dune] O'Caml's build system uses an S-expression based configuration language.
+[Shake][shake]'s build configuration language is [Haskell](https://www.haskell.org/), in its full generality.  This makes it hard to write tooling to modify `Shake` configuration files.
+
+[Dune][dune], O'Caml's build system, uses an S-expression based configuration language.

  [dune]: https://dune.build/
  [make]: http://www.crufty.net/help/sjg/bmake.html

BuildAutomation modified by Joseph Koshy

Joseph Koshy — Sun, 29 Dec 2019 19:20:15 -0000

--- v14
+++ v15
@@ -1,48 +1,52 @@
 # Build Automation

-This wiki page describes a proposed build system for the [Elftoolchain project](Home).  Some aspects of its design are briefly touched upon.
+This wiki page describes a proposed build system for the [Elftoolchain project](Home), briefly touching upon some aspects of its design.

 ## Goals

 The goals of this system are:

--   To implement repeatable & hermetic (i.e., fully deterministic) builds from source.
--   To allow developers to securely share build cycles with each other (peer-to-peer sharing of compilation cycles).
--   To offer a more expressive build language than [make][make].
+-   To implement repeatable & hermetic (i.e., fully deterministic) and verifiable builds from source.
+-   To allow developers to securely share build cycles with each other (peer-to-peer sharing of their build).
+-   To offer a more expressive build language than [make][make] that is also tooling friendly.
 -   To support cross-building of source code.
 -   To be frugal in its machine resource usage.

 ### Repeatable Builds
 **Local Repeatability**

-When a build is *locally repeatable* two runs of `make` (or equivalent build tool) on the same source tree would result in exactly the same bits being generated.  Examples of local non-determinism introduced into a build includes the presence of build timestamps in generated output, the presence of pathnames to temporary files in generated output, output differences caused by unstable orderings of inputs, and the like.
-
-Prior work in making builds locally reproducible includes:
+For the purposes of this document, a build is said to *locally repeatable* when two attempts at building the same source tree would result in exactly the same bits being generated.  Many compilation processes do not guarantee local repeatability out of the box.  Local non-deterrminism may be caused by the presence of timestamps in the output of a build step, embedded temporary filenames in generated output, output differences caused by unstable orderings of inputs, the use of special macros like `__DATE__` and `__TIME__`, and the like.
+
+Prior work in making builds locally repeatable includes:

 -  [Reproducible Builds](https://reproducible-builds.org/docs/) by the [Debian](https://www.debian.org/) project.
 -  [Reproducible Builds in FreeBSD](https://people.freebsd.org/~emaste/2017-03-12-AsiaBSDCon-Reproducible-Builds-FreeBSD.pdf) from the [FreeBSD](https://www.freebsd.org/) project.
+-  [Deterministic Builds](https://blog.torproject.org/deterministic-builds-part-one-cyberwar-and-global-compromise) at the [Tor Project](https://www.torproject.org/).

 Some Elftoolchain tools support locally repeatable output (usually specified using the `-D` option).

-A (locally) repeatable build allows the build process to be treated as a *pure function* mapping a set of source artifacts to a set of built artifacts.  Local repeatability would be the first step towards sharing build cycles across multiple developers.
+A locally repeatable build allows the build process to be treated as a *pure function* mapping a set of source artifacts to a set of built artifacts.  Local repeatability would be the first step towards sharing build cycles across multiple developers.

 **Cross-Machine Repeatability**

-Local repeatability is not sufficient - we need the same output to be generated from a given source tree when built on two different machines.  Sources of such "non-local" non-determinism would include the use of embedded (absolute) pathnames to source code, dependencies on the time zone of the build machine, and so on.
+Local repeatability is not sufficient - we need the same output to be generated from a given source tree when built on multiple machines.  Such "non-local" non-determinism would be caused by embedded (absolute) pathnames in generated output, dependencies on the time zone of the build machine, and so on.
+
+For the purposes of this document a build is said to be *cross-machine repeatable* if two builds on two different machines result in identical outputs.

 ### Hermetic Builds

-In a *hermetic build* every input to the build process is fully specified.  This includes:
+In a *hermetic build* every input to the build process would be fully specified.  These inputs include:

 - The version of the compilation tools in use.
-- The configuration used when invoking tools: invocation flags, linker scripts, and so on.
-- Specifying each and every module that a given step in the build depends on.  This includes the headers, libraries and configuration traditionally considered to be part of the "base system".
+- The configuration used when invoking compilation tools: invocation flags, linker scripts, and so on.
+- Each and every module that a given step in the build depends on.  This includes the headers, libraries and configuration traditionally considered to be part of the "base system".
+- And of course the notional "source code" being built.

 A hermetic build process guarantees that the build does not depend upon inputs that are "external" in some fashion (and are thus uncontrolled).

-One way to achieve hermeticity would be have the build framework invoke the individual build tools for each build step in an controlled environment that only contains the files that were explicitly determined to be inputs for the build step (e.g., a [chroot](https://en.wikipedia.org/wiki/Chroot), [jail](https://www.freebsd.org/doc/handbook/jails.html) or a symlink tree to a set of read-only inputs).
-
-If the build configuration is tracked in version control alongside the rest of the source tree, and if the source code to the compilation tools is also part of the source tree (e.g., [NetBSD](https://www.netbsd.org/docs/guide/en/part-compile.html)) then the build can be made *hermetic through time*.  With a hermetic-through-time build we would be able to roll back a source tree to a prior revision and be able to build exactly the same output as would have been originally built at that revision.
+One way to achieve a hermetic build would be have the build framework invoke the individual steps in a build in an controlled environment that contains precisely those files that were explicitly determined to be inputs for the build step (e.g., the build step would be run in a [chroot](https://en.wikipedia.org/wiki/Chroot), a [jail](https://www.freebsd.org/doc/handbook/jails.html) or a symlink tree to a set of read-only inputs).
+
+If build configuration is tracked in version control alongside the rest of the source tree, and if the source code to the compilation tools is also part of the source tree (e.g., as in [NetBSD](https://www.netbsd.org/docs/guide/en/part-compile.html)) then the build can be made *hermetic through time*.  With a hermetic-through-time build we would be able to roll back a source tree to a prior revision and be able to build exactly the same output using the exact same tooling as would have been originally built at that revision.

 In this document, a build that is both cross-machine repeatable and hermetic is called a *fully deterministic* build.

@@ -52,33 +56,43 @@

 ![Peer to peer builds](https://elftoolchain.sourceforge.io/images/p2p-builds.jpg)

-Sharing builds in this fashion would help speed up builds of large source trees (e.g., the equivalent of a `make world` or a run of `build.sh`).  The build utility would choose between either performing a local deterministic build from sources, or retrieving the already built artifact for that build step from its peers at each build step (or occasionally doing both, for verification).
-
-The peer-to-peer build system could be made independent of the build front end. This would both ease development, and would permit the peer-to-peer network to be potentially used by multiple build front ends (e.g., [Bazel][bazel] or [Buck][buck], in addition to proposed build tool).
+Sharing builds in this fashion would help speed up builds of large source trees (e.g., the equivalent of a `make world` or a run of `build.sh`).  Our build utility could choose between either performing a local fully deterministic build from sources, or retrieving the already built artifact for that build step from its peers (or occasionally doing both, for verification).
+
+The peer-to-peer build system could be made independent of the build front end, possibly implementing an existing API such as the [Bazel remote execution API](https://github.com/bazelbuild/remote-apis).  This would both ease development, and would permit the peer-to-peer build network to be potentially used by multiple build front ends (e.g., [Bazel][bazel] or [Buck][buck], in addition to proposed build tool): 

 Related work: [Bazel][bazel] supports caching of build results using a (centralized) [remote cache](https://docs.bazel.build/versions/master/remote-caching.html).

-### Herd Immunity
-
-Peer-to-peer builds could also be used to implement a form of "herd immunity"; any output artifacts that are tampered with would be flagged by the build network. This process could be used to verify, in a decentralized fashion, that the final artifact generated by a build (say the ISO image containing an OS installer) had actually been built from a specific revision of its source tree. 
-
-The design of this system would need to preserve correctness in the presence of large players (with significant amounts of compute power at their disposal) attempting to game the system.
+### Herd Immunity And Peer Verifiability
+
+Peer-to-peer builds could also be used to implement a form of "herd immunity".  Once we have multiple builders building the same source tree and sharing metadata about their builds, it would become possible to detect output artifacts that had been tampered with.
+
+This tamper detection protocol could be used to verify, in a decentralized fashion, that the final artifact generated by a build (say the ISO image containing an OS installer) had actually been deterministically built from a specific revision of the OSes source code.
+
+For the purposes of this document, a build system that can offer an auditable trail of its build steps is said to be*auditable*. An auditable build process that is coupled with peer-to-peer verification of its build steps is termed a *peer verifiable* build process. 
+
+### Cross-Organisational Verifiability
+
+Sometimes a software build would take for its input one or more binary blobs built by external organisations, for which source code is not available.
+
+Ordinarily the presence of such binary blobs would preclude the build process from offering a meaningful guarantee of the integrity of the final product.  Nevertheless, if the binary blob can be published along with a certificate that confirms that the blob had been verifiably built (or peer-verifiably built) from its source code, then that certificate could possibly be made part of the audit trail for the current build, improving the guarantee offered for its final output.
+
+For the purposes of this document, a build process which has the ability to incorporate such an external certificate into its audit trail is said to offer *cross-organisational verifiability*.
+
+TODO(jkoshy): Look for existing standards and protocols implementing cross-organisational verifiability, if these exist.

 ### The Build Configuration Language

-We need a more expressive build configuration language than what [make][] currently offers.
-
-#### Separation of concerns
-
-The  configuration used to specify build & test scenarios (i.e., `-c optimize` vs `-c debug` vs running tests under `valgrind`) should be separate from the configuration that specifies the dependencies between input and output artifacts.
-
-We want users to be able to define their own build and test scenarios easily.
-
-#### Tooling friendliness
-
-The build configuration language should be tooling-friendly. Large projects almost always need such tooling support.
-
-This means that the build configuration language should be easy to parse and modify programmatically.
+BSD operating systems are built from source using [BSD make][make] (e.g., the [FreeBSD makeworld](https://www.freebsd.org/doc/handbook/makeworld.html) build process, or by using an additional tool that drives [make][], such as in [NetBSD's build.sh](https://www.netbsd.org/docs/guide/en/part-compile.html)).
+
+Unfortunately, [make][]'s specification language is not expressive enough to implement a fully-deterministic build system of the type desired.
+
+This section lists some of the traits desired in a build specification language.
+
+At this point of time I am inclined to use a notation that is close to S-expressions, for the reasons outlined in some of the sections below.
+
+#### Tooling friendliness of the notation
+
+The build configuration language should be tooling-friendly. Large projects almost always need such tooling support. This means that the build configuration language should be easy to parse and modify programmatically.

 [Bazel][] and [Buck][] use a syntax reminiscent of Python for some aspects of their build rules although other aspects, such as build dependencies, are specified using a different (embedded) syntax. Build extensions for these systems are written using a Python-like extension environment; please see [Starlark](https://docs.bazel.build/versions/master/skylark/language.html) for [Bazel][], and [Buck Macros](https://buck.build/extending/macros.html) for [Buck][].

@@ -88,7 +102,7 @@
     name = "foo",
     srcs = ["foo.c", "bar.c"],
     hdrs = ["foo.h"],
-    deps = ["//some/package"],
+    deps = ["//some/package"],  # dependencies use a different sub-language
   )
 ```

@@ -112,7 +126,13 @@

 [Make][], in contrast uses a single global namespace, with its attendant lack of modularity.  Large projects then end up using recursive invocations of `make`, which all the drawbacks documented in  [Recursive make Considered Harmful](http://aegis.sourceforge.net/auug97.pdf) (Peter Miller, 2008).

-#### Multiple logically connected outputs per directory
+#### Separation of concerns
+
+We would like the configuration used to specify build & test scenarios (i.e., different compilation modes: `-c optimize` vs `-c debug` or running tests under `valgrind` vs running them normally) to be separate from the configuration that specifies the dependencies between input and output artifacts.  These two aspects of build configuration are intertwined in [make][].
+
+We want users to be able to easily define their own scenarios for building and testing code.
+
+#### Multiple logically connected build outputs per directory

 The BSD `/usr/share/mk/*.mk` framework currently works best when a single sub-directory in a source tree builds one program or one library. This restriction can be fairly limiting.

@@ -122,21 +142,25 @@

 Builds for large projects would need a language for writing "build macros" and the like.

+[make][]'s language is quite limited.  [BSD make][make] allows certain rules to be designated as "macros", by using the `.USE` pseudo target as a dependency.  [BSD make][make] also offers iteration using `.for`/`.endfor` and conditional execution using `.if`/`.endif`.  However these and other extensions to the base "make" syntax do not fit together well, which means that specifying large systems with [make][] remains awkward.
+
 [Bazel][] and [Buck][] have their own "macro" definition languages; these are distinct from the build configuration language itself.

 If we go the S-expression route, then a declarative subset of R*n*RS Scheme may be a natural choice for the extension language.  

+### Other Features Desired For The Build System
+
 #### Offline Operation

 The build system should be able to work in 'offline' mode, without needing external network access to function (although builds would be entirely local in this case).

-This feature would be useful when developers are frequently "off grid".
+This feature would be useful for developers in areas of poor network connectivity.

 #### Frugality

 The build system should be able to work effectively in a resource constrained host environment (such as a laptop computer).

-[Bazel][] in particular seems to need a lot of memory when building large projects.
+[Bazel][] in particular seems to need a lot of memory when building projects.

 #### Cross-platform Builds

@@ -148,9 +172,15 @@

 The build system should be able to work efficiently on a source tree with a small number of local changes, sharing the build of the unmodified parts of the source tree with its peers.

-#### Portability
-
-New target OSes should be easy to support, either through cross compilation, or by invoking the necessary build steps "remotely" using a small, portable helper that runs on the target OS.
+#### Easing (Build) Portability
+
+New target architectures should be easy to support, either through cross compilation, or by invoking the necessary build steps "remotely" using a small, portable helper that runs on the target hardware.
+
+#### Support for Bootstrapping
+
+The build tool should be usable when bootstrapping an operating system from source.
+
+This means that a substantial part of the build system needs to be written in portable C, so that it can be built and can run in a cross-hosted OS bootstrap environment.

 ## Sources

@@ -168,13 +198,11 @@
  [bazel]: https://bazel.build/
  [buck]: https://buckbuild.com/

-These two systems implement *Applicative* builds.
-
-We would need the full generality of *Monadic* builds; *monadic* build systems allow the structure of the build graph to be determined by the output of a previous build step.
-
 #### Monadic vs Applicative Build Systems

-Please see ["Applicative vs Monadic build systems"](http://neilmitchell.blogspot.com/2014/07/applicative-vs-monadic-build-systems.html) by Neil Mitchell for a discussion of the differences between the two kinds of build systems.
+[Bazel][] and [Buck][] implement *Applicative* builds.
+
+Looking over some of the steps involved in building BSD operating systems from source, it appears that we would need the full generality of *Monadic* builds; *monadic* build systems allow the structure of the build graph to be determined by the output of a previous build step. Please see ["Applicative vs Monadic build systems"](http://neilmitchell.blogspot.com/2014/07/applicative-vs-monadic-build-systems.html) by Neil Mitchell for a discussion of the differences between the two kinds of build systems.

 Open-source build systems that are monadic in nature include:

BuildAutomation modified by Joseph Koshy

Joseph Koshy — Sun, 29 Dec 2019 11:01:35 -0000

--- v13
+++ v14
@@ -1,23 +1,50 @@
 # Build Automation

-This wiki page describes a proposed build system for the [Elftoolchain project](Home).
+This wiki page describes a proposed build system for the [Elftoolchain project](Home).  Some aspects of its design are briefly touched upon.

 ## Goals

 The goals of this system are:

--   To implement deterministic, repeatable builds from source.
--   To allow developers to share build cycles with each other.
+-   To implement repeatable & hermetic (i.e., fully deterministic) builds from source.
+-   To allow developers to securely share build cycles with each other (peer-to-peer sharing of compilation cycles).
 -   To offer a more expressive build language than [make][make].
+-   To support cross-building of source code.
 -   To be frugal in its machine resource usage.

-### Deterministic Builds
-
-Deterministic builds allow the build process to be treated as a *pure function* mapping a set of source artifacts to a set of built artifacts. 
-
-Once the build process is a 100% deterministic, we can leverage this to share build cycles across multiple developers. 
-
-Note: some Elftoolchain tools support deterministic output (usually specified using the `-D` option).
+### Repeatable Builds
+**Local Repeatability**
+
+When a build is *locally repeatable* two runs of `make` (or equivalent build tool) on the same source tree would result in exactly the same bits being generated.  Examples of local non-determinism introduced into a build includes the presence of build timestamps in generated output, the presence of pathnames to temporary files in generated output, output differences caused by unstable orderings of inputs, and the like.
+
+Prior work in making builds locally reproducible includes:
+
+-  [Reproducible Builds](https://reproducible-builds.org/docs/) by the [Debian](https://www.debian.org/) project.
+-  [Reproducible Builds in FreeBSD](https://people.freebsd.org/~emaste/2017-03-12-AsiaBSDCon-Reproducible-Builds-FreeBSD.pdf) from the [FreeBSD](https://www.freebsd.org/) project.
+ 
+Some Elftoolchain tools support locally repeatable output (usually specified using the `-D` option).
+
+A (locally) repeatable build allows the build process to be treated as a *pure function* mapping a set of source artifacts to a set of built artifacts.  Local repeatability would be the first step towards sharing build cycles across multiple developers.
+
+**Cross-Machine Repeatability**
+
+Local repeatability is not sufficient - we need the same output to be generated from a given source tree when built on two different machines.  Sources of such "non-local" non-determinism would include the use of embedded (absolute) pathnames to source code, dependencies on the time zone of the build machine, and so on.
+
+### Hermetic Builds
+
+In a *hermetic build* every input to the build process is fully specified.  This includes:
+
+- The version of the compilation tools in use.
+- The configuration used when invoking tools: invocation flags, linker scripts, and so on.
+- Specifying each and every module that a given step in the build depends on.  This includes the headers, libraries and configuration traditionally considered to be part of the "base system".
+
+A hermetic build process guarantees that the build does not depend upon inputs that are "external" in some fashion (and are thus uncontrolled).
+
+One way to achieve hermeticity would be have the build framework invoke the individual build tools for each build step in an controlled environment that only contains the files that were explicitly determined to be inputs for the build step (e.g., a [chroot](https://en.wikipedia.org/wiki/Chroot), [jail](https://www.freebsd.org/doc/handbook/jails.html) or a symlink tree to a set of read-only inputs).
+
+If the build configuration is tracked in version control alongside the rest of the source tree, and if the source code to the compilation tools is also part of the source tree (e.g., [NetBSD](https://www.netbsd.org/docs/guide/en/part-compile.html)) then the build can be made *hermetic through time*.  With a hermetic-through-time build we would be able to roll back a source tree to a prior revision and be able to build exactly the same output as would have been originally built at that revision.
+
+In this document, a build that is both cross-machine repeatable and hermetic is called a *fully deterministic* build.

 ### Shared Peer To Peer Builds

@@ -25,13 +52,17 @@

 ![Peer to peer builds](https://elftoolchain.sourceforge.io/images/p2p-builds.jpg)

-Sharing builds in this fashion would help speed up builds of large source trees (e.g., the equivalent of a `make world` or a run of `build.sh`).  The build utility could choose for each build step between either performing a deterministic local build from sources, or retrieving the already built artifact for that step from its peers (or both, for verification).
-
-The peer-to-peer build system would be made independent of the build front end. This would ease development, and would permit the peer-to-peer network to be potentially used by multiple build front ends (e.g., [Bazel][bazel] or [Buck][buck], in addition to our proposed build tool).
+Sharing builds in this fashion would help speed up builds of large source trees (e.g., the equivalent of a `make world` or a run of `build.sh`).  The build utility would choose between either performing a local deterministic build from sources, or retrieving the already built artifact for that build step from its peers at each build step (or occasionally doing both, for verification).
+
+The peer-to-peer build system could be made independent of the build front end. This would both ease development, and would permit the peer-to-peer network to be potentially used by multiple build front ends (e.g., [Bazel][bazel] or [Buck][buck], in addition to proposed build tool).
+
+Related work: [Bazel][bazel] supports caching of build results using a (centralized) [remote cache](https://docs.bazel.build/versions/master/remote-caching.html).

 ### Herd Immunity

-Peer-to-peer builds could also be used to implement a form of "herd immunity"; any output artifacts that are tampered with would be flagged by the build network. This process could be used to verify, in a decentralized fashion, that the final artifact generated by a build (say the ISO image containing an OS installer) had actually been built from a specific revision of its source tree.  The design of this system would need to preserve correctness in the presence of large players (with significant amounts of compute power at their disposal) attempting to game the system.
+Peer-to-peer builds could also be used to implement a form of "herd immunity"; any output artifacts that are tampered with would be flagged by the build network. This process could be used to verify, in a decentralized fashion, that the final artifact generated by a build (say the ISO image containing an OS installer) had actually been built from a specific revision of its source tree. 
+
+The design of this system would need to preserve correctness in the presence of large players (with significant amounts of compute power at their disposal) attempting to game the system.

 ### The Build Configuration Language

@@ -77,7 +108,7 @@

 The build system should support the structuring of project code into logical modules.

-Build configuration should also be modular: users should be able to write their build files without need to worry about conflicts between their symbols and any symbols defined elsewhere.
+Build configuration should also be modular: users should be able to write their build files without need to worry about conflicts between the symbols used in their configuration files and any symbols defined elsewhere.

 [Make][], in contrast uses a single global namespace, with its attendant lack of modularity.  Large projects then end up using recursive invocations of `make`, which all the drawbacks documented in  [Recursive make Considered Harmful](http://aegis.sourceforge.net/auug97.pdf) (Peter Miller, 2008).

BuildAutomation modified by Joseph Koshy

Joseph Koshy — Sat, 09 Nov 2019 22:52:20 -0000

--- v12
+++ v13
@@ -49,19 +49,19 @@

 This means that the build configuration language should be easy to parse and modify programmatically.

-[Bazel][] and [Buck][] use a Python-like syntax for their build rules. 
-
+[Bazel][] and [Buck][] use a syntax reminiscent of Python for some aspects of their build rules although other aspects, such as build dependencies, are specified using a different (embedded) syntax. Build extensions for these systems are written using a Python-like extension environment; please see [Starlark](https://docs.bazel.build/versions/master/skylark/language.html) for [Bazel][], and [Buck Macros](https://buck.build/extending/macros.html) for [Buck][].
+ 
 ```python
   # Defines the build process for library "libfoo".
   cc_library(
     name = "foo",
     srcs = ["foo.c", "bar.c"],
     hdrs = ["foo.h"],
-    deps = [...],
+    deps = ["//some/package"],
   )
 ```

-We could adopt this syntax, or alternately use a notation that is based on S-expressions. For example, using [SRFI-88/89](https://srfi.schemers.org/srfi-88/srfi-88.html)-esque syntax:
+We could adopt this syntax (for continuity), or we could alternately use a notation that is based on S-expressions which may be easier to extend.  For example, using an [SRFI-88/89](https://srfi.schemers.org/srfi-88/srfi-88.html)-esque syntax:

 ```scheme
  ; Defines the build process for library "libfoo".

BuildAutomation modified by Joseph Koshy

Joseph Koshy — Wed, 23 Oct 2019 22:00:08 -0000

--- v11
+++ v12
@@ -89,9 +89,9 @@

 #### Expressivity

-Builds for large projects would need a language that is [expressive](https://en.wikipedia.org/wiki/Expressive_power_(computer_science)) than "plain old data", e.g., for writing "build macros" and the like.
+Builds for large projects would need a language for writing "build macros" and the like.

-An "extension language" would need to be part of the build system.  [Bazel][] and [Buck][] have their own "macro" definition languages; these are distinct from the build configuration language itself.
+[Bazel][] and [Buck][] have their own "macro" definition languages; these are distinct from the build configuration language itself.

 If we go the S-expression route, then a declarative subset of R*n*RS Scheme may be a natural choice for the extension language.