Re: [Cucumber-linux-development] New Buildscript Format

[LM <lm...@gm...>]

On Mon, May 7, 2018 at 6:04 PM, Scott Court <sc...@cu...> wrote:
> The original buildscript format was heavily inspired by the Slackware
> .SlackBuild format.

I started experimenting with the SlackBuild format several years ago.
I like it better than many other Linux build systems, but one
immediate drawback was that if something changed in the scripts, you
had to change it by hand everywhere.  There were some nice tools that
helped create the script in the first place, but the tools didn't
modify a created script if a change needed to be made globally.

> It doesn't use any form of templates. This means it is not possible to make
> any sort of global change without manually editing every single buildscript.
> It doesn't sufficiently track information about the package's upstream. For
> example, it would be very helpful if the build system could track (amongst
> other things):

There are different ways to factor out the common code so that it's
not repeated by hand in each build script.  One way is similar to what
Arch Linux does, where there's a compiled tool that does most of the
work and you just need to supply the basic information to build the
package.  The other way is to use a scripting language instead of a
compiled program.  Arch Linux users like that their system is fast and
efficient.  If a program is compiled, it's typically going to be more
efficient than a scripting language.  However, if you're changing
something often or debugging, it's much easier to work with a
scripting language.  My personal choice with the build system I'm
currently using is to go with a scripting language.  I felt the
flexibility was more important than the speed for my particular
situation.  I expect things to change often and I don't expect them to
stabilize any time soon.

If one goes with a scripting language, one also needs to decide which
scripting language to use or if you want to come up with a syntax that
does not tie you down to any particular scripting language.  Slackware
uses shell/bash style scripting.  BSD make systems typically use BSD
make ( http://bsdbuild.hypertriton.com// ).  I've seen some projects
use JavaScript for scripting.  My original idea was to not tie my
system down to any particular scripting language.  So, one would use a
common syntax that can generate the desired scripting language using
templates.  However, bash or sh are available on most platforms I work
with and I ended up using some less portable syntax in certain cases
and working mainly with bash/sh as the scripting language of choice
similar to Slackware Slackbuilds.  I could change the decision in the
future, but it would take some modifications to the individual build
scripts.

I also knew I wanted to use a template based system, so the next step
after deciding on a language was to decide what tool to use for
working with/updating templates.  As a C/C++ programmer, one of the
more familiar template systems is just using a preprocessor.  However,
I absolutely wanted the ability to concatenate variables and be able
to interpret the results of the concatenation as another variable.  I
could reinvent the wheel and write my own template engine, but there
are already several out there.  I ended up using gpp (
https://files.nothingisreal.com/software/gpp/gpp.html ).  It's very
similar to a C preprocessor with similar syntax.  However, it does a
few things a C preprocessor isn't capable of doing.  It doesn't have
all the functionality I would like in an ideal situation, but it has
the minimum I needed.

Using a template based system, I still end up creating one build
script per package (similar to a Slackbuild script).  I find it
convenient for sharing build scripts to be able to just give someone
the actual script and it has everything they need to complete the
build.  With a system like Arch, you'd need to supply the build tools
and the package information (partial script) to recreate a build.

I also tried to modularize everything as much as possible.  It's not
the easiest code to read, but I basically use the preprocessor and
macros like one might use functions in a programming language.  I try
to factor out repeated code.  If I find I need to put the same exact
information in each package's build script, then I know I'm doing
something wrong.  Repeated information should be part of the template
code or kept globally somewhere.

> What the upstream URL is for a project.

The MXE project uses a clever way to track when a project is updated.
It isn't maintenance free, but it makes it easier to track when an
upstream project changes than to have to check every project by going
to each URL and looking for changes.  They do a wget on the URL where
information on the project version resides and parse the version
number out.  If the version doesn't match the version of the package
they're expecting, they know it's time to look into updating the build
script.  There's an example of one of their build scripts including
the update code here:
 https://github.com/mxe/mxe/blob/master/src/curl.mk

> What the signature/checksum of the source tarball is.

I think most build systems include ability to validate checksums
including what MXE and ARCH use.  With my own system, I can specify
what type of checksum I'm using for a specific package, (sha1sum,
sha256sum, etc.).  That way, if a project supplies some type of
checksum, I can use whatever they do.

The way I know if I'm on the right track with my design is by seeing
how fast it can adapt to changes.  When I update a package, if it's
just a minor update it may just take changing the version number,
regenerating and rerunning the build script and it'll fetch the new
version of the source code and build.  Major changes to build a
package may take several hours or even a few days.  However, they'd
take as long if I wasn't using a build script and it would be much
harder to recreate what I'd done or give the steps to someone else to
reproduce.  I recently changed my build system to handle
cross-compiling in order to build packages for Android.  It took at
least a few days to add all the features I wanted.  However, once I
updated my build scripts, build time for producing an Android apk went
down significantly.  Also, my system was flexible enough to adapt to
the new situation even if it did require some major modifications in a
few areas.

I went through a lot of pros and cons when designing a build system
for my own use and I did a lot of investigating what other build
systems did to see if I could use them or ideas from them and not
completely reinvent the wheel.  It would have been nice to discuss the
pros and cons of various build system designs with others, but I
really didn't have any luck finding other developers who were
interested in that sort of thing.  I'll be very interested to read
about the design decisions that go into the new Cucumber Linux build
system.  I did try to have a clear idea of my main design goals before
starting.  I also did a lot of research on the various trade-offs of
design decisions.  I'm sure what I decided wouldn't necessarily be the
most popular choices, but I went with what was practical and would
work best for my situation.

I look forward to reading how the Cucumber Linux build system progresses.

Sincerely,
Laura