Re: [Cucumber-linux-development] New Buildscript Format

[Scott C. <sc...@cu...>]

On 05/08/2018 08:37 AM, LM wrote:
> On Mon, May 7, 2018 at 6:04 PM, Scott Court <sc...@cu...> wrote:
>> The original buildscript format was heavily inspired by the Slackware
>> .SlackBuild format.
> I started experimenting with the SlackBuild format several years ago.
> I like it better than many other Linux build systems, but one
> immediate drawback was that if something changed in the scripts, you
> had to change it by hand everywhere.  There were some nice tools that
> helped create the script in the first place, but the tools didn't
> modify a created script if a change needed to be made globally.
>
>> It doesn't use any form of templates. This means it is not possible to make
>> any sort of global change without manually editing every single buildscript.
>> It doesn't sufficiently track information about the package's upstream. For
>> example, it would be very helpful if the build system could track (amongst
>> other things):
> There are different ways to factor out the common code so that it's
> not repeated by hand in each build script.  One way is similar to what
> Arch Linux does, where there's a compiled tool that does most of the
> work and you just need to supply the basic information to build the
> package.  The other way is to use a scripting language instead of a
> compiled program.  Arch Linux users like that their system is fast and
> efficient.  If a program is compiled, it's typically going to be more
> efficient than a scripting language.  However, if you're changing
> something often or debugging, it's much easier to work with a
> scripting language.  My personal choice with the build system I'm
> currently using is to go with a scripting language.  I felt the
> flexibility was more important than the speed for my particular
> situation.  I expect things to change often and I don't expect them to
> stabilize any time soon.
I'm planning use a scripting language instead of a compiled program for
those reasons, and also because of the increased portability that comes
with using a script instead of a compiled program. I also think that
having a build system that's easy to work with, modify and port is more
important than having one that runs quickly.
> If one goes with a scripting language, one also needs to decide which
> scripting language to use or if you want to come up with a syntax that
> does not tie you down to any particular scripting language.  Slackware
> uses shell/bash style scripting.  BSD make systems typically use BSD
> make ( http://bsdbuild.hypertriton.com// ).  I've seen some projects
> use JavaScript for scripting.  My original idea was to not tie my
> system down to any particular scripting language.  So, one would use a
> common syntax that can generate the desired scripting language using
> templates.  However, bash or sh are available on most platforms I work
> with and I ended up using some less portable syntax in certain cases
> and working mainly with bash/sh as the scripting language of choice
> similar to Slackware Slackbuilds.  I could change the decision in the
> future, but it would take some modifications to the individual build
> scripts.
All of the buildscripts for Cucumber Linux 1.x are already written in
Bash, so it would probably be easiest to stick with that.
> I also knew I wanted to use a template based system, so the next step
> after deciding on a language was to decide what tool to use for
> working with/updating templates.  As a C/C++ programmer, one of the
> more familiar template systems is just using a preprocessor.  However,
> I absolutely wanted the ability to concatenate variables and be able
> to interpret the results of the concatenation as another variable.  I
> could reinvent the wheel and write my own template engine, but there
> are already several out there.  I ended up using gpp (
> https://files.nothingisreal.com/software/gpp/gpp.html ).  It's very
> similar to a C preprocessor with similar syntax.  However, it does a
> few things a C preprocessor isn't capable of doing.  It doesn't have
> all the functionality I would like in an ideal situation, but it has
> the minimum I needed.
One approach I'm thinking of is to turn the repeated sections into Bash
functions and effectively make a Bash API for the buildsystem. The
*.buildinfo files would then source this API and use the Bash functions.
My main concern with this approach is its impact on readability, since
it would effectively replace easy to understand Bash commands with
function calls that a developer would then have to look up. This issue
can be at least partially mitigated though by commenting all of the
functions calls well.

The other option is of course to use a preprocessor like you said, but
then that would require one to re-preprocess the buildscript every time
a change is made to it. I'm worried that someone (i.e. me) may forget to
do this at some point and break stuff as a result. It also adds another
dependency to the buildsystem (the preprocessor). This approach does
result in a more readable final result (if done properly); however, it
is not editable in it's final form: you still have to edit it in terms
of macros.

From an editing standpoint it looks like they're similar. Like you said
the macros behave similarly to functions in this case. I think I'm
leaning more towards the Bash API though since it removes the need to
re-preprocess. Also, if a change to the API is made with the Bash
approach, there's no need to re-preprocess anything; you just have to
replace all the old API bash source files with the new ones and call it
a day. With a preprocessor, it would be necessary to re-preprocess every
single buildscript whenever this happens.

Your thoughts?
> Using a template based system, I still end up creating one build
> script per package (similar to a Slackbuild script).  I find it
> convenient for sharing build scripts to be able to just give someone
> the actual script and it has everything they need to complete the
> build.  With a system like Arch, you'd need to supply the build tools
> and the package information (partial script) to recreate a build.
That is one minor drawback of the buildscript-buildinfo approach
(https://github.com/cucumberlinux/new-buildscript-format/tree/master/buildscript-buildinfo-format):
it requires you to distribute two scripts instead of one. That being the
case though, it's already necessary to redistribute the slack-desc and
doinst.sh files with each buildscript in order for it to be usable
anyway. So I'm not sure adding another file is a huge issue since you
already have to distribute more than one.
> I also tried to modularize everything as much as possible.  It's not
> the easiest code to read, but I basically use the preprocessor and
> macros like one might use functions in a programming language.  I try
> to factor out repeated code.  If I find I need to put the same exact
> information in each package's build script, then I know I'm doing
> something wrong.  Repeated information should be part of the template
> code or kept globally somewhere.
Agreed. I definitely want to make that sort of stuff global. That's what
I was thinking with the API.
>> What the upstream URL is for a project.
> The MXE project uses a clever way to track when a project is updated.
> It isn't maintenance free, but it makes it easier to track when an
> upstream project changes than to have to check every project by going
> to each URL and looking for changes.  They do a wget on the URL where
> information on the project version resides and parse the version
> number out.  If the version doesn't match the version of the package
> they're expecting, they know it's time to look into updating the build
> script.  There's an example of one of their build scripts including
> the update code here:
>  https://github.com/mxe/mxe/blob/master/src/curl.mk
>
>> What the signature/checksum of the source tarball is.
> I think most build systems include ability to validate checksums
> including what MXE and ARCH use.  With my own system, I can specify
> what type of checksum I'm using for a specific package, (sha1sum,
> sha256sum, etc.).  That way, if a project supplies some type of
> checksum, I can use whatever they do.
We could definitely use something like that. Currently, I track most
packages by hand and it can be quite tedious at times, so thanks. I'll
add implementing something like this to the To Do List. Not a top
priority for now, but adding the version/URL/checksum/signature
information to the new buildsystem will make this substantially easier
to implement in the future.
> The way I know if I'm on the right track with my design is by seeing
> how fast it can adapt to changes.  When I update a package, if it's
> just a minor update it may just take changing the version number,
> regenerating and rerunning the build script and it'll fetch the new
> version of the source code and build.  Major changes to build a
> package may take several hours or even a few days.  However, they'd
> take as long if I wasn't using a build script and it would be much
> harder to recreate what I'd done or give the steps to someone else to
> reproduce.  I recently changed my build system to handle
> cross-compiling in order to build packages for Android.  It took at
> least a few days to add all the features I wanted.  However, once I
> updated my build scripts, build time for producing an Android apk went
> down significantly.  Also, my system was flexible enough to adapt to
> the new situation even if it did require some major modifications in a
> few areas.
>
> I went through a lot of pros and cons when designing a build system
> for my own use and I did a lot of investigating what other build
> systems did to see if I could use them or ideas from them and not
> completely reinvent the wheel.  It would have been nice to discuss the
> pros and cons of various build system designs with others, but I
> really didn't have any luck finding other developers who were
> interested in that sort of thing.  I'll be very interested to read
> about the design decisions that go into the new Cucumber Linux build
> system.  I did try to have a clear idea of my main design goals before
> starting.  I also did a lot of research on the various trade-offs of
> design decisions.  I'm sure what I decided wouldn't necessarily be the
> most popular choices, but I went with what was practical and would
> work best for my situation.
>
> I look forward to reading how the Cucumber Linux build system progresses.
>
> Sincerely,
> Laura
Thank you, glad to have your expertise.

    - Scott