Re: [Reduce-algebra-developers] How to build REDUCE reproducibly?

[Arthur N. <ac...@ca...>]

> SOURCE_DATE_EPOCH. ...
I now collect a date from the "$Id:" record that subversion updates on a
file version.h when that is checked in. I then check in using a script 
"scripts/commit.sh" that tends to make a frivolousx change to version.h so 
that subversion will check it in alongside all other changes.

> Thank you for mentioning the address space randomization, I didn't know about 
> that. But as I mentioned I observe exactly the same executable files reduce, 
> bootstrapreduce, and csl from build to build. Does it mean that if any 
> address randomization is present, it all happens at runtime, not buildtime?

I believe that happens as code is loaded, is well after build time and 
just before the user code is run - and it may have an effect on where 
"new" or "malloc" places data as well as where executable code is loaded 
to. The object is to make attacks that are based on knowing addresses with 
the code harder to set up. But I do not know full internal details.

> Problem with build paths nowdays is considered to be solved by simply 
> building in predictable paths. In can be just a fixed path,

That is a relief. If I am LUCKY it will turn out to be possible to avoid 
build path sensitivity, but I can now push that well down the priority 
list!

>> Well image files go through a compression process so changes in content - 
>> even if small - may lead to a different file-size. 
> That is interesting. I think that compression shutoff would be important step 
> on the path to clarification. Is it hard to switch it off?

It is a scheme that is coded in, so it is not a matter of there being a 
switch. It is hack the code to disable it. Well there will only be a few 
entrypoints it all goes through so on some sense it would be easy!

However when I was implementing the stuff that codes and decodes image 
files I had a scheme that could trace what was going in in semo-readable 
form and awakening that is a matter of "-DDEBUG_SERIALIZE=1" or some such.
Also since the executable files are identical (so far) I can use just a 
fixed version that and just run the strep that remakes .img files
several times. And if when I find divergences compare the trace files - 
which are HUGE.

==== (later) ====

I enabled serialization traces and also to make that at all feasible I set 
up to build a sort of mini-reduce image. And changed bits of the Reduce 
internals to avoid storing some other things in heap images.

I *believe* I may have sorted it. The issue is that the Reduce supervisor 
wants to support a statement "time;" that reports how much time has been 
used - and for that it reads the clock as it starts up and saves that 
value even if you do not than ask for a time measure. That starting clock 
number was ending up in the heap image. If was often a small integer and 
differing values dis not alter the image size. And SOMETIMES it would 
report the same value on consecutive startups so sometimes there could be 
agreement.  This behaviour impacts bootstrapreduce.img. I can sort it out 
by zeroing out those vars before saving an image in the build job.

I have not done proper testing yet (it is TEDIOUS) but have good hopes.
Now reduce.img will certainly be impacted by this so my changes will 
improve things there but there may be yet more to do. We are creeping 
again towards end-of-day so that is not for just now. I may just get to
where I can check in a version where bootstrapreduce.img is built cleanly 
- you mentioned a case where it had "just one" bit of variation.

So this part of it was "clock" in that it was variation in the time the 
system spent getting from when it started up as far as what is in effect
    base_clock := read_clock();

Arthur


>> And this is 
>>> presumably EITHER a case where something is still sensitive to the clock 
>>> or or
>>> sensitive to memory layout that can change with Linux memory address 
>>> randomization.
> Sometimes I have observed that two separate builds performed on different 
> days are exactly the same, including all the images. I think this allows to 
> exclude the "clock" possibility.
>
> Best regards,
> Andrey Ignatenko
>