zpaqfranz 65.5

This is marked as pre-release, because there is a lot of new code to debug

BTW a even bigger 65.6 is coming 😄

Finally: the wiki is (more or less) aligned

Four headlines:

• -m8: zstd inside zpaq. Fast as the fast methods, smaller than -m1, and still extractable by any zpaq, even 7.15.
• -m6 is now the "4.5": the -m4 you know, plus three cheap contexts that take about 40% of the gain of -m5 for a quarter more time. LZ4 and LZAV (the old -m6 and -m7) are gone.
• a -append rewritten: two passes, the archive is written only forward (never a seek back), and -append -stdout sends the new version to a pipe as a delta you can glue after the archive.
• autotest -all -quick: a four-step self test for the slow machines (NAS), and the NAS builds use all their cores on ARM.

This document has three parts:

• Part one: what is new, in short, for everybody
• Part two: the same things in more detail, for power users
• Part three: "lo spiegone", how it works inside, and why, for developers

---

Part one, what is new, for the user

1. -m8: zstd
2. -m6: the "4.5"
3. a -append: only forward, and to stdout
4. <[inline_block>0](#4
5. NAS: all the cores
6. Smaller things

1. -m8: zstd

zpaqfranz a z:\backup.zpaq c:\data -m8
zpaqfranz a z:\backup.zpaq c:\data -m8 -turbo

zstd 1.5.7 is built into zpaqfranz. On the test machine (Ryzen 7950X3D, RamDisk, Windows), with -turbo:

	-m1	-m8
VM, 11.7 GB, time	15.4 s	6.5 s
VM, archive	5,641 MB	5,440 MB
116,703 files, 6.5 GB, archive	1,407 MB	1,317 MB

Smaller than -m1, more than twice as fast on a big file. The archive is a normal zpaq archive: zpaqfranz decompresses it with native zstd, any other zpaq (7.15 included) with a zstd decoder stored inside the archive itself.

-m8h19 is stronger (zstd level 19), -m86 uses 64 MB blocks.

2. -m6: the "4.5"

zpaqfranz a z:\backup.zpaq c:\data -m6

Many asked for something between -m4 and -m5. -m6 is now exactly -m4, with three more "sparse" contexts in its CM path: on the same two test sets it is 1.2% (a VM) and 4.3% (many small files) smaller than -m4, for 24% more time. -m5 goes further (2.9% / 10.7%) but needs 3 to 5 times the time, to compress and to extract.

The old -m6 (LZ4) and -m7 (LZAV) are gone: -m8 beats both in size at the same speed. Archives made with them are still extracted (the decoder is inside the archive). -m7 has no method of its own any more: today it goes through the same code as -m5.

3. a -append: only forward, and to stdout

zpaqfranz a z:\backup.zpaq c:\data -append
zpaqfranz a /backup/data.zpaq /data -append -stdout >/tmp/delta.zpaq

-append (the "anti-ransomware" mode, for files that can only grow, as chflags sappend on BSD) now really writes the archive only forward: it reads the data twice, the first time only to know the sizes of the header. The two passes must see the same files, folders and bytes, otherwise nothing is kept and the exit code is 2.

With -stdout the archive on the command line is only read. If it does not exist, stdout gets a complete archive; if it exists, stdout gets only the new version, to be glued after it:

cat /backup/data.zpaq /tmp/delta.zpaq > /backup/new.zpaq
copy /b z:\data.zpaq+z:\delta.zpaq z:\new.zpaq

Always check the exit code: a stream with an exit code other than 0 is not valid.

4. autotest -all -quick

zpaqfranz autotest -all -quick -to /tmp/qt
sh /tmp/qt/quicktest.sh

The full autotest -all needs 18 GB and hours on a NAS. The quick suite does exactly four things: extracts the built-in sha256.zpaq, compresses the data with -m1, then with -m8, then tests both archives. Its final check fails if one of the four is missing, or if anything else was run.

5. NAS: all the cores

The NAS builds (and every ARM Linux build) used 1 thread whatever the CPU: now a 4 core ARMv8 NAS uses 4 threads (32 bit builds: 2, as always for 32 bit).

6. Smaller things

• -mx... and -ms... are the short form of -method (-mx6.0ci1am). Before they were "Unknown option ignored", and the archive was made, in silence, with -m1.
• mount on Windows as Administrator: the drive letter is created for the whole system (Mount Manager), so Explorer sees it.
• The Makefile has ENABLE_ZPAQMOUNT=yes to build mount (FUSE found with pkg-config, with fallbacks for OpenBSD, macOS, the BSDs).
• mount in a build without it (no -DZPAQMOUNT) now says "This zpaqfranz has NO mount command" with exit code 2, instead of the generic help with exit code 0. A build with mount shows +M after the JIT in the banner.
• monitor (Win64) works also in the builds without SFTP (it was wrongly inside #ifdef SFTP: the command was accepted and not run).
• The help lists zip and pakka on every platform (they always worked everywhere, but the help showed them only on Windows); the help of k says that -to is mandatory.
• A batch of fixes of issues reported on GitHub (see part three).

---

Part two, the details, for power users

1. -m8 in practice
2. -m6: what exactly changed
3. <[inline_block>0](#3
4. <[inline_block>0](#4
5. <[inline_block>0](#5
6. The NAS builds

1. -m8 in practice

switch	meaning
-m8	zstd level 3, 16 MB blocks (like -m1)
-m8hN	zstd level N (1..22)
-m8aN	zstd "fast" level -N
-m86...	64 MB blocks (the digit after the 8 is the block size, as in the other levels)

• The whole block is one zstd frame, the window is the block: no checksum (zpaq has its own SHA-1 for every fragment).
• -m8 goes well with -turbo: on a big file the limit is the fragmenter, not the compressor, and with -turbo -m8 reaches the speed of -m0.
• autotest checks at start that the ZPAQL decoder built into zpaqfranz is the frozen one (a SHA-1 is compared): a build where it changed says so and must not be released.
• Not in the builds for old compilers (-DANCIENT without -DNAS, -DESX): there -m8 is not available. The NAS builds have it.

2. -m6: what exactly changed

-m6 is level 4, with one difference in the CM path chosen by the analysis of the block:

m4   x6,<e8>ci1,1,1,1,2a[w]m
m6   x6,<e8>ci1,1,1,1,2ac0,2,0,255c0,3,0,0,255c0,4,0,0,0,255[w]m

Everything else is the same as -m4: the pre-check of compressibility, the choice between store, LZ77, BWT and CM, E8E9 when there is x86 code, the word model on text, 64 MB blocks. Standard zpaq components: any zpaq extracts it, no new decoder.

Measured (a CM on a Ryzen 7950X3D, 16 cores):

	VM 11.7 GB	vs -m4	time	116,703 files	vs -m4	time
-m4	4,888,062 KB		409 s	885,432 KB		244 s
-m6	4,827,398 KB	-1.2%	509 s	847,422 KB	-4.3%	302 s
-m5	4,744,893 KB	-2.9%	1921 s	790,770 KB	-10.7%	810 s

Extraction costs as compression (a CM must redo the same work).

3. a -append

• Pass one: the normal add(), the same scan, the same fragmenter, the same compression, but nothing is written and nothing is opened for writing. It gives the sizes of the header and three counters: files added, folders added, bytes read from the files.
• Pass two: the header is written at once with its final sizes, then the data. No seek back.
• The check: same counters and same sizes, else the new version is removed (the archive is truncated back, or deleted if it was new) and the exit code is 2. It is a check by quantity: a file replaced, between the two passes, by another one of the same size is not seen.
• -stdin is refused (it cannot be read twice).
• A problem in pass one (a file that cannot be read, a damaged archive) stops everything before writing: exit code of pass one.
• BSD and macOS: the archive is opened in append mode (ab+), so a chflags sappend file can be updated (the normal a cannot). Something after the last good version (an incomplete transaction) is refused there: trim it first.
• With the same data and the same -timestamp, a and a -append give the same archive, to the byte (checked with -m1 and -m8, new and existing archives, on Linux, FreeBSD, OpenBSD, macOS, Windows).

4. a -append -stdout

• The archive on the command line is never created, written or truncated: if it exists it is only read.
• Not there: stdout gets a complete archive (the same bytes a would write).
• There: stdout gets only the new version, a delta. cat archive delta gives the same archive of a normal a on a copy, to the byte.
• -key: the delta is encrypted with the same key and the same salt of the archive (it has only one), and does not start with a new salt. A wrong key, or no key (there is no prompt: stdout is the archive), is refused before sending a byte.
• A damaged or truncated archive is refused before sending a byte.
• Only archive bytes on stdout (messages on stderr or silent). On Windows stdout is binary.
• Not with: multipart archives, -chunk, -index, -franzen, -fasttxt.
• -stdout without -append is unchanged.

5. autotest -all -quick

step	what
1	x sha256.zpaq and sum -rename (the 256 files)
2	a quick_m1.zpaq te -m1
3	a quick_m8.zpaq te -m8
4	t quick_m1.zpaq, t quick_m8.zpaq

The script quicktest.sh (quicktest.bat on Windows) ends with autotest -all -quick -checktxt: every step writes its number in quickops.txt, and the check wants 1, 2, 3, 4 and the expected outputs, and nothing else (another number, an output of the full suite: failure). -quick keeps its general meaning everywhere else. The quick suite does not need the 18 GB of the full one.

6. The NAS builds

• -DNAS is for modern machines with little RAM, -DANCIENT for really old ones: the limits of NAS (buffers, no SFTP, no mount, no HW SHA...) are the same as before.
• The number of threads on Linux came from the "cpu cores" line of /proc/cpuinfo, which exists only on x86: every ARM got 1 thread. Now, without that line, every "processor" line is a core. Measured in qemu: Cortex-A53 build 4 threads (was 1), ARMv7 build 2 (was 1; 32 bit builds stay at 2 at most). A real -DANCIENT build is unchanged.

---

Part three, lo spiegone, for developers

1. zstd and -m8: how, and at what price
2. The "4.5": how it was found
3. -append: the double pass
4. -append -stdout: a stream that only goes forward
5. The quick suite
6. Fixes of GitHub issues

1. zstd and -m8: how, and at what price

How

• zstd 1.5.7, the official amalgamation (combine.py), without multithreading and dictBuilder, compiled as C++ inside namespace zzstd (zpaqfranz has its own U32, U64, xxhash...). 759 macros are saved and restored around it (push_macro/pop_macro) so they do not leak into the rest of the source.
• A block is stored with a PCOMP: the ZPAQL program that decodes a zstd frame (about 910 lines of ZPAQL, generated by a script). zpaq 7.15, or any zpaq, runs it and gets the data back. zpaqfranz recognizes that exact PCOMP (byte by byte) and calls native zstd instead.
• The encoder: window = the whole block, no checksum, content size in the frame. The native decoder: ZSTD_decompressStream.

The good

• Size: smaller than -m1 (-3.6% on a VM, -6.4% on many files), much smaller than LZ4 (-14% / -36%) and LZAV (-9% / -23%).
• Speed: with -turbo the fast methods all reach the speed of -m0 on a big file (the fragmenter is the limit): -m8 6.5 s against 15.4 s of -m1.
• Extraction: native zstd, the fastest of all.
• Compatibility: the archive is still a zpaq archive, every zpaq extracts it.

The bad

• The source got big. zpaqfranz.cpp went from 148,378 lines (5.2 MB, 65.4m) to 188,125 lines (6.9 MB, 65.5g): about 47,600 lines are zstd. It compiles slower, and a "single file you can read" is less readable.
• The PCOMP is forever. Once archives exist, the ZPAQL decoder inside them cannot change: a bug there stays in those archives. And zpaqfranz never runs it in normal use (it uses native zstd): a bug would be seen only by zpaq 7.15 (or p -verify). This is why it was tested hard (every zstd level, frames of every kind, about 30,000 frames made by decodecorpus, 20,000 damaged streams, extraction with zpaq 7.15), and why autotest checks its SHA-1. A different decoder, one day, will be a new PCOMP next to this one, never a change of this one.
• ZPAQL is slow: when zpaq 7.15 runs the PCOMP, 80-95 MB/s with the JIT. Usable, not fast.
• Old compilers: gcc 3.4 (ESXi) has no push_macro: -m8 is not in the -DESX and -DANCIENT builds.

2. The "4.5": how it was found

The obvious ideas were wrong. Adding the "clever" parts of -m5 at the end of -m4 (SSE, MIX2, a 16 bit MIX) gave almost nothing: they work on the output of the models, and if the models do not see a structure, mixing their guesses does not create it.

The method: on a sample (16 slices of 64 MB along a VM disk), start from -m4 and add one piece of -m5 at a time; start from -m5 and remove one piece at a time. Both said the same thing: the piece that matters is the three sparse contexts c0,2,0,255, c0,3,0,0,255, c0,4,0,0,0,255 (removing them from -m5 cost 8.5 MB, everything else together less).

A sparse context looks at the byte 2, 3 or 4 positions back, and at the position modulo 2, 3 or 4: the high byte of a 32 bit integer is predicted by the high byte of the previous integer, not by the byte just before it (the low one, almost random). Binary data are full of that: integers, pointers, tables, structures, RGB pixels.

Why they cost so little: in -m5 each of them is followed by an ISSE of order 1, tables of 2^19..2^20 rows, 32-64 MB each, a random access for every bit (cache misses). Without the ISSE they are ICMs of 2^13..2^14 rows, 0.5-1 MB: they stay in cache. That keeps 8.5 of the 9.6 MB of their gain, at the speed of -m4.

The candidate was then tested with a build where the CM path of level 4 had them, so the pre-check of -m4 stays (a hand written x... method skips it, and works hard on data already compressed). That candidate is now -m6: it does not reach -m5, and it is not meant to; it makes -m4 a bit better for a small price.

3. -append: the double pass

• append() runs add() twice. Pass one has g_fakewrite: the OutputArchive opens nothing and only moves a logical offset. At the end it stores the compressed size and the fragment table size.
• Pass two (g_appendpass2) writes the jidac header with those sizes and skips the final "seek to the header and write it again".
• The counters: files added (files_added + files_updated), folders written in the index (counted in buildindexchunks), bytes read (total_done). They, and the sizes, must be the same in both passes: a header with the wrong size would make the archive unreadable.
• On a mismatch, after the close: the archive is truncated back to the start of the new version (or deleted if it was new; the chunks of this run deleted with -chunk), exit code 2.
• The date: both passes use the same one (it is taken once at start, or from -timestamp), so the two passes compress the same bytes.
• Checked by tracing the descriptor of the archive (strace, truss, ktrace, dtruss): the normal a seeks back to the header after the data, a -append never does.

4. -append -stdout: a stream that only goes forward

• OutputArchive gets a stream mode: fp is stdout (the handle on Windows, so no text translation), off is the logical position inside the archive; seek accepts only the current position (or, before the first byte, the start of the new data), tell returns off + ptr.
• Encryption: the AES counter is the logical position, so a delta that starts at the end of an existing archive decrypts after cat. The salt comes from the existing archive; a new archive starts with a new one.
• Nothing after the close touches the archive on the command line (no trim, no date, no -fasttxt); control-C has nothing to roll back.
• A tail after the last good version (the archive size is not the end of the last transaction) is refused: the delta would be glued after garbage.

5. The quick suite

One table (g_autotestquick) makes both the script and the final check: each entry has its commands, its -out files and what they must contain. The script writes echo N >>quickops.txt after each operation. The check reads the numbers (1..4, in order, nothing more), greps the outputs, and fails on any quickNN.txt not in the table or any outNN.txt of the full suite. Removing the -m8 operation from the script, adding a fifth number, or dropping an out03.txt in the folder: each one fails the check.

6. Fixes of GitHub issues

A batch of fixes of issues reported on GitHub, one by one reproduced (on Linux and on Windows), fixed, tested:

292, 293, 294, 295, 296, 297, 298, 299, 300, 301, 302, 303, 304, 305, 306, 307, 308, 309, 310, 311, 312, 313.

Not everything was changed as asked in the issues: some requests were declined on purpose, and one (the SIGINT handler of 308, which is not async-signal-safe) needs a redesign and is not done yet.