NGI0-Entrust-SDCC

NLnet is funding an NGI0 "SDCC" project done by @spth and @felixs. NLnet manages the NGI0 Entrust fund, which was established with financial support from the European Commission's Next Generation Internet programme, under the aegis of DG Communications Networks, Content and Technology under grant agreement No 101069594.

Monthly progress reports

January: Usually, SDCC does yearly releases near the beginning of the year. 4.3.0 will be next. To prepare for the 4.3.0 release, I worked on bug reports this month. The project has a few 20-hour "bug sprints" that I can use for this. At first, it felt more like wading through mud than a sprint, though: the very first issue I worked on, [bugs:#3533], took 7 hours. Then, progress became faster though: for some bugs there were patches by users that I did review and apply, for others a single underlying issue turned out to have resulted in multiple open tickets. For a while, an uphill hike in the mountains replaced the sprint feeling: [bugs:#3539] could only be reproduced on 32-bit big-endian systems, so I dug out my old Mac mini G4 running OpenBSD. It is about as slow as a Raspi 3, but still better than nothing. While it compiled and ran regression tests during many days/nights, I worked on other issues.
In total, I worked on bugs for 44 hours in January, closing the following tickets: [bugs:#3533], [bugs:#3504], [patches:#453], [feature-requests:#859], [bugs:#3539], [patches:#449], [patches:#454], [patches:#442], [patches:#456], [patches:#455], [patches:#435], [bugs:#3532], [bugs:#3523], [bugs:#3487], [bugs:#3505], [bugs:#3474], [bugs:#3503], [support-requests:#185], [support-requests:#182], [bugs:#3525], [bugs:#1885], [bugs:#2608], [bugs:#2738], [bugs:#2878], [bugs:#2928], [bugs:#2933], [bugs:#2983], [bugs:#2998], [bugs:#3012], [bugs:#3019], [bugs:#3020], [bugs:#3024], [bugs:#3048], [bugs:#3054], [bugs:#3058], [bugs:#3103], [bugs:#2802], [bugs:#3191], [bugs:#3539], [bugs:#3516], [bugs:#3398], [bugs:#3400], [bugs:#3394], [bugs:#3275], [bugs:#3228], [bugs:#3263], [bugs:#3407], [bugs:#3160], [bugs:#3070], [bugs:#3023], [bugs:#3252], [bugs:#3025], [bugs:#3497], [bugs:#3249], [bugs:#3399], [bugs:#2249], [bugs:#2279].

February: I was quite busy with other aspects of my life, so there was not much progress. A few of the issues that needed dealing with were in areas that @felixs is more familiar with than me, so we agreed that he'd do one or two of the 20-hour bug sprints. Myself, I only worked on bugfixes for 20 hours in February, closing the following tickets: [bugs:#3542], [feature-requests:#868], [patches:#457], [bugs:#3543], [bugs:#3060], [bugs:#3545], [bugs:#3547], [bugs:#2887], [bugs:#3352], [bugs:#3548], [bugs:#3549], [bugs:#3550], [bugs:#2776], [bugs:#3378], [bugs:#3382], [patches:#459], [bugs:#3560].

March: The beginning of the month wasn't much different from the months before. I worked 16h on bugfixes, closing [patches:#319], [bugs:#3328], [bugs:#3290], [bugs:#3167], [bugs:#3564], [bugs:#3556], [bugs:#3567]. Towards the end of the month, my focus shifted to the f8. Now, the f8 port can compile, assemble and link all regression tests and passes some.

April: At the beginning of the month I continued work on the f8. Now all regression tests pass (when using the default compiler options - some thing still fail when using different optimization settings). To be able to do slightly more realistic comparisons to other ports I started implementing basic peephole optimization. To be able to do a bit of benchmarking on the simulator later, I soon want to implement a basic timer. Towards the end of the month I went back to fixing SDCC bugs to prepare for the 4.3.0 release. I worked on SDCC issues for 11h, closing [bugs:#3566], [bugs:#3570], [bugs:#3030], [bugs:#3574], [bugs:#3576], [bugs:#3575]. Again I was also busy with parts of my life other than this project. In April those other parts included both personal matters and fixing issues in stm8flash, a tool which is often used together with the stm8 port of SDCC.

May: At the beginning of the month, I mostly worked on the f8. Later, preparations for the 4.3.0 release took over again: fixing bugs, and implementing _BitInt support for hc08, s08 and mos6502, so it will be ready for 4.3.0 in all maintained ports. In total, I worked on bugfixes for 38h, closing the following tickets [bugs:#3579], [bugs:#2874], [bugs:#3585], [bugs:#3559], [bugs:#3455], [bugs:#3348], [bugs:#3590], [bugs:#3591], [bugs:#3140], [bugs:#3593]. @felixs also continued work on preprocessor-related bugs. Now, trunk is frozen, and a release next month looks likely. Meanwhile, work on new features started in the next branch, to be merged to trunk after 4.3.0: so far, mostly optimizations for multiplications, checked integer arithmetic and bit rotations.

June: There were some further delays in the SDCC 4.3.0 release process related to the Windows installers. But we now have a release candidate 1 for SDCC 4.3.0, and so far no release-critical bugs have been found.
Meanwhile, in the next branch work continued on optimizations for rotations, bugfixes and some code refactoring. Despite 23 hours of bugfixing, the only ticket closed is [bugs:#3596], since the fixes for other bugs are in the next branch only: they didn't make it in time for the 4.3.0 code freeze.
In the genconstprop branch, I started work on generalized constant propagation. By now, many important use-cases are already covered, and code gets optimized. There are multiple phases: Analysis, machine-independent optimizations and machine-dependent optimizations. In analysis, for every local variable, at every point in the program, we keep some information on the possible values: an interval and a bitmask. In machine-independent optimizations, the information is used to replace operations by cheaper ones in iCode (the intermediate three-address code used in SDCC). E.g. It turned out that one task planned in the original memorandum-of-understanding for this NGI0 project (optimizing the standard library manually by making use of bit-precise integer types) could be replaced by an optimization based on generalized constant propagation that automatically narrows variables to a suitable unsigned bit-precise integer type. The machine-dependent optimizations make use of the information from analysis in code generation and register allocation, and are implemented in the individual backends.

July: Finally, we made the [SDCC 4.3.0 Release]. Meanwhile, in the genconstprop branch, generalized constant propagation was extended to be able to not just handle integers, but also pointers, which is very useful for backends that have a non-uniform address space, such as mcs51 and pdk: knowing to which intrinsic named address space a pointer points at an access can save a lot of cycles (we can then just generate the correct instructions instead of having to do it through a helper function that first looks at the upper address bits to then branch to the appropriate instructions). Soon after the 4.3.0 release, the next branch got merged to trunk, resulting in some bug fixes, and thus closing of tickets [bugs:#3595], [feature-requests:#876], [bugs:#805], [bugs:#3603], [bugs:#881], [bugs:#3606], [bugs:#3604], [bugs:#3605], [bugs:#3609], [bugs:#3359]. A few days later the genconstprop branch followed. As usual (some people try release candidates, but many use a release), some bugs were found after the release, some of which (along with some older ones) were then fixed in trunk, closing tickets [patches:#465], [bugs:#3619], [bugs:#3559], [bugs:#3626], [bugs:#3597], [bugs:#2345], [bugs:#2240], [bugs:#2744] in 25 hours of bugfixing. However, current trunk is still not as stable as it was at the time of the release. In particular, there is a bug that results in silent generation of wrong code that I have not yet been able to track down: It is sporadic, and the only host on which I see it is Debian on aarch64, which means running experiments on a (relatively slow) Raspi4, and not being able to get the incorrect asm a second time (it looks like I get the problem about once in a thousand runs of sdcc, and maybe only during high system load).

August: In August, I mostly worked on the f8. On one hand writing a (still incomplete) proof-of-concept Verilog implementation, on the other hand refining instruction set details, and changing/fixing the f8 assembler and simulator. I did also work on fixing bugs in SDCC., in particular on the sporadic bug already mentioned for July. While trying to track it down using UB sanitizers, I did find and fix some UB in SDCC, but not the bug I was trying to fix. A few other tickets got closed also (thanks to patches provided by a user): [bugs:#3636], [bugs:#3638], [bugs:#3639] in 16 hours of bugfixing. I'll try to get a faster aarch64 system in September, and hope that this will help me track down the issue. Besides the aarch64 issue, there is also a not-yet-resolved mcs51 codegen bug affecting variables of types wider than 32 bits that looks realease-critical to me. IMO, both need to be fixed before a 4.4.0 release. I hope we can release SDCC 4.4.0 in January 2024.

September: I was quite busy with other aspects of my life, and used only a bit of time for SDCC. The sporadic bug on Debian GNU/Linux on aarch64 is still there, and I don't have a faster aarch64 machine for debugging it yet. But I found and fixed a bit more UB in SDCC, and also worked on other issues. In particular, I did build a powerpc64 machine, which will be very useful for debugging, as some bugs are much easier to detect and reproduce on big-endian machines. I looked into bugs for 20h and also improved optimizations based on generalized constant propagation. Few tickets got closed: [patches:#467], [bugs:#3648], [bugs:#3649]. The powerpc64 machine will be put into the SDCC compile farm in early October, to help catch any endianness bugs. I'll also contact NLnet to make some minor changes in the MoU (to reflect that part of the work is being done by @felixs, and to account for changes in the plan, in particular generalized constant propagation turning out to be so powerful, that it removed the need for some manual tuning of standard library functions).

October: We ran out of in-project bugfixing hours months ago, so there is not much point continuing to record those. Of course I still fix bugs, but for quite a while that bugfixing hasn't been part of this project, so starting with this report I won't list any details or hours for that. The powerpc64 machine is in the compile farm now (running GNU/Linux, not the originally intended OpenBSD, which turned out to be far too slow), and the GNU/Linux aarch64 machine will soon be replaced by a faster one.
Finally, what I believe to have been the last serious issues in generalized constant propagation have been fixed; snapshots build, regression test results look good. We now see a lot more green dots on https://sdcc.sourceforge.net/snap.php than at the beginning of the month. There are a few serious bugs left that needed fixing before we can do the next major release, but I am optimistic about progress. Meanwhile, the focus of this project will shift back to the f8.

November: This month I didn't find much time to work on the project. Basically, I did some small improvements to generalized constant propagation (fixed a bug that resulted in some optimizations on bit-wide operations being omitted, implemented optimizations for code generation for shifts for some backends). By far most of the work I did on SDCC this month was outside this project (bugfixing to prepare for the 4.4.0 release, setting up the new aarch64 machine and getting it into the compile farm, improved code speed optimizations for the z80 and related ports). I hope to find time to really get back to the f8 work early next month.

December: Sorry for the late update. There was no real progress on the NGI0 project in December. Instead the focus was on working towards the SDCC 4.4.0 release with bugfixes; by 2023-12-21, we had SDCC 4.4.0 RC1.

January: A few more issues in the release candiates were found and fixes, but we finally released SDCC 4.4.0, finally bringing all the optimizations and bugfixes I worked on since mid-2023 to SDCC users . Otherwise, there wasn't much progress on this project, though.

February: I mostly worked on the Verilog model of the single-cycle f8 core. It is now reasonably complete, each instruction works in a small test program with the core simulated in Icarus. It was nearly there already last year, but the last few instructions still took quite some effort. Actually, most of my time was spend on fixing interrupt handling.

March: Due to its mature free toolchain, I went with a Lattice iCE40 as first target; when I needed more memory, I looked into the CologneChips GateMate. The f8 Verilog code now synthesizes for the iCE40UP5K and GateMateA1 FPGAs. Post-synthesis tests pass for the iCE40UP5K. For the GateMateA1, there are some test failures, but it looks like this might be a problem in yosys, and I'm in contact with the authors of the synth_gatemate pass in yosys to look into it. Two simple demos (LED blinking and "Hellor, world!" output via software-emulated UART) work on both the iCEBreaker and GateMateA1-EVB boards. However, there are still serious issues: Dhrystone on the GateMateA1-EVB crashes after the first three lines of output. Also, this is my very first nontrivial Verilog design, and it shows. It is big and slow (about 5 MHz on the iCE40UP5K). In retrospective, going for a single-cycle core probably wasn't a good choice for the f8 with its many read-modify-write instructions. The single-cycle approach required a program memory with multiple read ports (which the iCE40 doesn't have), which now looks like too much to demand for an 8bit architecture, that is meant to fill the gap below 32-bit CPUs,such as RISC-V. For now, my f8 implementation is both bigger and slower than a minimal RISC-V.
This was the last month of this project. For the f8, there is already a follow-up project: [NGI0-f8], giving me a chance to do better.