Re: [Freedos-devel] Interim Build Delayed

[C. M. <pu...@ul...>]

On at 2023-09-01 19:03 +0200, Bernd Böckmann via Freedos-devel wrote:
> Well, I think it's me to blame ;-)
> 
> I will try to build some compression mechanism into AMB, so that the 
> help files get smaller. The main FreeDOS help file would also benefit 
> from that, I think.
> 
> Bernd

I recently added one of my executable compression formats to my 
debugger's online help, allowing it to "assemble" (build) the help pages 
at build time and then compress them before inclusion (by NASM incbin 
directive) into the main assembly pass. Perhaps you want to copy parts 
of this.

I used the heatshrink library, "A data compression/decompression library 
for embedded/real-time systems." [1]. It is available under the ISC 
License [2]. I created two different depackers, both under the Fair 
License (but feel free to use and copy these under any usage 
conditions). One is the executable depacker, which can deal with data 
(both compressed and decompressed) exceeding 64 KiB, using 8086 
segmented addressing [3]. The other is a bit smaller and assumes that 
all data fits in a single segment each, though the source and 
destination can be in different segments [4].

For the recent lDebug release 6 I did not yet enable these build 
options, although everything is implemented for the in-memory online 
help to be largely compressed. I'm compressing each help page as its own 
data block, unrelated to one another. Despite this, almost all of the 
data does compress down by 20% to 60%. (In heatshrink use, the 
percentage indicates how much of the uncompressed size is dropped to 
reach the compressed size.)

The depacker needs about 300 bytes in my application (for only the 
memory-to-memory depacker). The compression is done in a loop at build 
time [5]. I try every -w parameter from 10 to 14 and every -l parameter 
from 4 to 14 (but -l must be below -w), and use the best result. 
(Usually -w 10 -l 4.) (I just found that the valid ranges for the 
parameters are actually larger than this, still testing how well those 
work.) I use a slightly modified compressed file format, in which the 
first byte is the -w parameter and the second byte is the -l parameter, 
and then the original compressed data file is appended. The depacker 
needs to know the packed size to know when to end depacking. In my 
debugger's depacker, I append a NUL byte at the end for the user to know 
how long the unpacked data is, but this could be modified easily to 
return the size.

Regards,
ecm


[1]: https://github.com/atomicobject/heatshrink
[2]: 
https://github.com/atomicobject/heatshrink/blob/7d419e1fa4830d0b919b9b6a91fe2fb786cf3280/LICENSE
[3]: https://hg.pushbx.org/ecm/inicomp/file/41860de1db0e/heatshr.asm
[4]: 
https://hg.pushbx.org/ecm/ldebug/file/654bf2cbfffb/source/hshrink.asm#l208
[5]: https://hg.pushbx.org/ecm/ldebug/file/654bf2cbfffb/source/mak.sh#l309