zpaqfranz 65.2

From 65.1a (2026-09-19) to 65.2k, one new command, a DEFLATE encoder written from scratch, and a long list of Windows things that used to fail on filenames nobody should be allowed to create in the first place.

The headline is zip: an archive can be turned into one single ZIP64 file, which any machine on this planet can open, with no zpaqfranz at the other end.

---

Part one, what is new, for the user

1. zip: the archive into one single .zip
2. zip -deflate: the same .zip, three times smaller
3. zip -all: one folder per version, the way mount shows them
4. Names Windows cannot create, fixed while extracting
5. rd: deletions that get to the end
6. mount: one engine, an ETA, a lot less RAM
7. zpaqfranz-full: WinFsp without downloading anything
8. Wildcards on *nix, patterns on <[inline_block>0](#8
9. Franzen-only archives
10. Smaller things

---

1. zip: the archive into one single .zip

zpaqfranz zip backup.zpaq

That is all. z:\backup.zpaq becomes z:\backup.zip, with everything the last version holds inside it.

zip is the x command: same filters, same -until, same -range, same everything. The only difference is where the bytes land, one single ZIP64 file instead of thousands of files on a filesystem.

The point is a restore file that travels. The Linux, NAS and Mac filenames that Windows cannot even create (: * ? " < > |, a trailing dot or space, CON, LPT1, paths longer than 255) are perfectly legal inside a zip. So the whole tree can be carried on a Windows box, or on an exFAT disk, handed over to somebody who has never heard of zpaq, and unpacked back on the original system exactly as it was.

zpaqfranz zip nas.zpaq -to z:\restore.zip              :: where you want it
zpaqfranz zip nas.zpaq /share/PUBLIC -to z:\pub.zip    :: only one folder
zpaqfranz zip nas.zpaq -until 100 -to z:\v100.zip      :: a specific version
zpaqfranz zip nas.zpaq -force                          :: overwrite the default name

Without -to the .zip is born beside the archive, same name, .zip extension, and in that case (and only in that case) it must not exist yet, unless -force.

What ends up in there: symlinks (as real symlinks, with their target as data), *nix permissions, the modification time with no timezone nonsense (Info-ZIP UT field, one second resolution), empty folders, and files bigger than 4 GB. Read-only by construction: the archive is never touched.

Nothing is compressed by default. The data inside a .zpaq is already compressed, unpacking it and packing it again costs time, and the STORED zip is what makes the whole thing cheap and fully multithreaded. If you do want it smaller, that is the next chapter.

2. zip -deflate: the same .zip, three times smaller

zpaqfranz zip sources.zpaq -deflate -to z:\sources.zip

Real numbers, measured on 19 files of Pascal and C++ sources, 5.78 MB:

zip              5.783.027 =>             2.141.820 (37.04%, 3.47 MB saved)

On zpaqfranz.cpp alone (4.68 MB) it gets 37.6%, where gzip -1 gets 35.1% and gzip -6 gets 30.0%. Throughput on one core is about 95 MB/s, everything included (zpaq decompression, CRC-32, compression and writing).

It is method 8, plain DEFLATE, the one thing every unzip on earth has been able to read since 1993. No new dependency: zpaqfranz does not link zlib and still does not.

It is meant for sources, text, logs, configuration files, mailboxes, anything that is still in its original form inside the archive. What is not worth it is left alone: folders, symlinks, files under 64 bytes, and the extensions that carry compressed data already (.zip .gz .7z .jpg .png .mp4 .mp3 .docx .xlsx .apk .jar and a few dozen more) stay STORED, so no CPU is wasted on them.

Even a file that looks compressible and is not cannot grow: on 200 KB of pure random data the entry came out 22 bytes longer than the original, not the 12% that a naive DEFLATE would have added.

-zip is an alias of -deflate, if you find it easier to remember.

3. zip -all: one folder per version, the way mount shows them

zpaqfranz zip backup.zpaq -all -to z:\every.zip

every.zip
   VER00000000\...   the archive as it was after the 1st add
   VER00000001\...   ... after the 2nd
   VER00000002\...   ... and so on

Exactly the folders the mount command shows, with exactly the same names and the same numbering, so VER00000002 in the zip is VER00000002 on the mounted drive and is what x -until 3 would write on disk.

Every folder is a full snapshot, not a delta: a file that never changed is in all of them, a file deleted in version 3 is in the first two folders and nowhere else. Which makes "give me the whole backup history in one file I can hand to the lawyer" a single command.

The price is written on the tin, because a zip does not deduplicate anything: an unchanged file is stored again in every single folder. So the damage is measured before anything is created:

zip  -all  1.349 version folder(s) (VER00000000 ... VER00001348)
zip  -all  32.534.075 entries  7.552.157.500.930 (6.87 TB), about 15.15 GB of RAM
71401! zip: -all, not enough free space. -range/-until to write less, -space to bypass

That took 50 seconds on a 1,349 version archive and stopped there, without eating the machine. Past 5 million entries it asks for -force as well.

-range is what makes it usable on a real archive: it picks which folders get written, while the versions before them are read anyway, because that is what a snapshot is made of.

zpaqfranz zip backup.zpaq -all -range 10:20 -to z:\10_20.zip
zpaqfranz zip backup.zpaq -all -range 1348  -to z:\last.zip

-deflate works together with -all, if you have the patience.

4. Names Windows cannot create, fixed while extracting

Extract a NAS backup on Windows and, until now, it died one file at a time with ERROR_INVALID_NAME (123), "the filename, directory name, or volume label syntax is incorrect". The names were perfectly legal where they were made and are forbidden here: < > " | ? * : and the backslash, the control characters, a name ending with a dot or a space, the DOS device names (CON, PRN, AUX, NUL, COM1...LPT9).

Now x (and w) fix them on the fly, by default, and say how many were changed. -nosanitize turns it off if you want the old behaviour.

The drive letter and the //?/ prefix survive the treatment, which is the part that used to go wrong: a long path stored inside the archive, extracted -to another long path, became //?/z:///?/z:/something and Windows refused it.

5. rd: deletions that get to the end

zpaqfranz rd z:\kajo -force -kill -space

Three changes, all of them about finishing the job.

It does not stop at the first failure. One locked file used to leave the folder half deleted, with no clue about which object was the culprit. Now everything else is deleted anyway, and at the end:

71345! NOT deleted     3
71346! First error     00000032 The process cannot access the file because it is being used by another process
71347! on <<z:/kajo/logs/current.log>>
71348: use -verbose to get every object that could not be deleted

Long paths and impossible names no longer need -longpath. Deleting does not need any compatibility with anything, so rd always goes through the extended //?/ syntax: a folder called acustica (with a trailing space), which cannot even be opened by a normal Win32 call, is now removed.

Junctions and directory symlinks are removed, never followed. This one mattered: walking into a junction meant erasing the files it points to, somewhere else on the disk. Now the link is removed and its target is left exactly as it was.

6. mount: one engine, an ETA, a lot less RAM

Much less memory on archives with many versions. A version used to be a full copy of the file table. With 746 versions of a 455,000 files NAS backup that is 340 million map nodes, tens of GB, std::bad_alloc long before the drive letter appeared. A version is now a point in the history of each file, and a file that never changes across 300 versions is stored once.

The long wait tells you how long. Building the directory trees of a 1,349 version archive used to print a line with no idea of the end:

 00:00:27  V00000367  sorting 72.871 files

and now:

  43.07%  00:00:40  eta 00:00:53  V00000581  sorting 72.871 files

One engine. The core engine (the memory-mapped scanner inherited from the standalone zpaqfuse project, plain archives only) is gone from the options: jidac reads everything it read, plus encrypted, Franzen and franzen-only archives. -backend auto and -backend jidac are the same thing now. The source of core has not been deleted, it is commented out line by line.

7. zpaqfranz-full: WinFsp without downloading anything

zpaqfranz-full.exe carries its dependencies inside itself and lays them down when they are needed. The WinFsp installer is now one of them.

So on a machine with no WinFsp, mount finds the MSI inside the executable, checks its SHA-256 and offers to install it, with no network at all:

WinFsp is not installed (or cannot be loaded)
winfsp-2.1.25156.msi extracted from this very executable (no download)
Installer verified (SHA-256 OK) C:\Users\...\Temp\winfsp-2.1.25156.msi

The plain zpaqfranz.exe keeps doing what it did before, downloading the pinned installer and verifying it.

8. Wildcards on *nix, patterns on dir

On Windows the shell expands z:\foo\nas* before zpaqfranz sees it. On Linux and BSD a quoted pattern arrived as it was written and matched nothing. Now it is expanded internally, the same way:

zpaqfranz a backup.zpaq "/tmp/nas*"

And the dir command applies a pattern to the name, not to the whole path, which is what you expect when you type

zpaqfranz dir /tmp/nas*

without having to fight with -only */nas*.

9. Franzen-only archives

An archive can now really be called pippo.franzen and have no cleartext .zpaq side at all. Everything that looked for <archive>.franzen now understands that, when the archive itself is the franzen file, there is nothing else to look for and the cleartext file must never be opened.

10. Smaller things

• Windows seek: fseeko() reported a failure that had happened somewhere else, long before. A seek to offset 0xFFFFFFFF and a real error look the same to SetFilePointer(), and the only way to tell them apart is to clear the error code first. Now it does.
• copy with several sources counted and copied the same file once per source. Fixed, and the error message now says which file it was about.
• Windows error messages are no longer cut at 30 characters when they are printed on their own (the padding is there to keep the columns aligned in the error report, and only there).
• x -recover, heavy testing and zip no longer print one line per file for "no stored hash" and "all zero hash": on 78,000 files that was hundreds of lines of scroll for something the summary says in one number. -verbose brings the list back.

---

Part two, technical detail, lo spiegone

1. zip

~1,000 lines before a single byte of DEFLATE, all of them free functions plus one struct, right before Jidac::extract(). The command letter is 'x': zip sets flagzip= true and goes through the very same extract(), which is why every filter, every -until, every -range works with no extra code.

Why STORED is not laziness

Everything is method 0, and that is what makes the extractor work unchanged: the compressed size IS the uncompressed size, and it is known before extracting anything. So zipprepare() walks dt once, computes the position of every local header and of every byte of data, checks the free space, writes all the local headers, and stores into each DT the absolute offset its data has to go to (zipdataoffset, zipindex).

After that the extractor keeps doing exactly what it does on N separate files: decompressThread seeks to offset + job.zipbase and writes. The zip is one file that is never closed between entries (g_zipfp), the write mutex it already had is enough, and the whole thing stays multithreaded.

The CRC-32 that nobody (re)computes

A zip entry needs the CRC-32 of the uncompressed data. Computing it again would mean reading everything a second time. It is not computed at all: the extractor already calculates the CRC-32 of every chunk it writes (g_crc32, the same machinery -checksum uses to verify), so zipfinalize() groups those chunks by filename, sorts them by position, and combines them with crc32_combine(). A hole that was never written (an all-zeros run the extractor skips) is combined in as crc32zeros(n).

Names, symlinks, attributes

zipmakename() turns a stored name into a legal zip name: backslashes become slashes, //?/ and //?/UNC/ prefixes go away, X:/something becomes X/something, leading slashes are dropped, and .. is neutralized in four passes so nothing can escape the extraction folder.

A symlink has no fragments at all inside the archive: the target sits in the posix block. In the zip it becomes an entry whose data is the target itself, with mode 0xA1FF (S_IFLNK | 0777) in the high 16 bits of the external attributes and "version made by" saying Unix, which is what unzip restores as a real symlink.

zipexternalattr() carries the *nix mode when the archive has one (attr & 255 == 'u'), or a sensible 0755/0644 when it does not, plus the MS-DOS directory bit. Dates go in twice: the MS-DOS pair (two seconds, no timezone, 1980 to 2107) and the Info-ZIP UT extra field with the real Unix time.

Zip64 is used per entry, on the uncompressed size, and for the end of central directory when there are more than 65,534 entries or the directory does not fit in 32 bits.

-deflate, the encoder

section: deflate, ~330 lines, no external library.

Minimal means fixed Huffman codes only (BTYPE=01): the tables are the ones printed in RFC 1951, so there is no tree to build, no code lengths to transmit and nothing to get wrong. zipdeflate_preparetabelle() builds them once, already bit-reversed, because deflate writes a Huffman code starting from its most significant bit and everything else from the least.

The matcher is a 3 byte hash ((b0<<16|b1<<8|b2) * 2654435761 >> 16), 64 K buckets, a prev[] chain, greedy, at most 16 candidates per position, no lazy matching. The positions a match jumps over are inserted into the table anyway, because that is where the next match usually is.

Input is compressed 256 KB at a time and no match ever crosses a chunk: it costs a whisker of ratio and it removes the sliding window, the rebasing of the hash table and every bug that lives in there. The bit stream, on the contrary, runs across the chunks, so what comes out is one single deflate stream closed by an empty final block.

While the tokens are built their exact cost in bits is summed. If the fixed block is not smaller than the same data in stored blocks (BTYPE=00, 5 bytes of header every 64 KB), the stored blocks are written instead. That is the bound: incompressible data costs 5 bytes every 64 KB, not the 12% that fixed Huffman codes would add to it.

s_deflate is built once and reused entry after entry with begin(FP): head[] 256 KB, prev[] 1 MB, tokens 1 MB, staging buffers. About 2.6 MB, once.

-deflate, why it needed another engine

The parallel extract() cannot do it. Its whole design is the precomputed layout described above, and a compressed entry does not know where it ends until it has ended.

The sequential engine of extractstdout() does exactly the opposite: it walks dt file by file and, for each file, its fragments in order, pulling the blocks from a cache that a pool of workers fills ahead of it. That is a stream, and a stream is what a deflate wants.

So zip -deflate goes there instead:

Jidac::extract()          arguments, read_archive, -all expansion, filters
   flagzip && flagdeflate -> extractstdout(0, "")   (dt is already loaded)
extractstdout()           worker pool, block cache, one file at a time
   zip_export_handler     header, deflate stream, crc, patch, central directory

extract_handler gained two virtual methods with an empty default implementation, begin_file() and end_file(), so the partition, raw and normal handlers do not even notice. In the main loop the change is three lines: open the entry, close the entry, and one else handler->write_fragment(...) where the file handle would have been.

zip_export_handler writes the local header with the CRC and the compressed size at zero, feeds the fragments to s_deflate, and when the file is over patches CRC, compressed size and uncompressed size back into the header (the 32 bit fields, or the 8+8 inside the zip64 extra, whose offset it remembered). Entries come out in dt order, which is already sorted by name, so the central directory needs no sorting.

zipgrande() decides zip64 on the uncompressed size plus the worst case a deflate can add to it, so a layout decided before compressing is still right after it. zipcsize() returns the uncompressed size when the method is 0, which is why the STORED path did not have to change a single line.

Files under 64 bytes, folders, symlinks and the compressed extensions (zipdeflate_inutile()) get method 0 and go straight out.

Since the data flows through the sequential engine, zip -deflate also verifies the stored hashes while it compresses, for free, which the parallel path does not do.

Validated by three independent decoders: Python zipfile (testzip()), Info-ZIP unzip -t and 7z t, on sources, mixed data and a 93 MB file whose SHA-256 after a round trip is the original one. Content, names, dates, attributes and CRCs of a deflated .zip are identical to the STORED one of the same archive.

-all, the version folders

read_archive(), when all > 0, already keys every entry it reads as 0001/name, one group per version, holding what that version did, a date of 0 being a deletion. That is the whole history.

zipexpandversions() replays it from the first version onwards and, at every step, pours the live state into VER%08u/, which is franzmount::mount_version_name() and is 0 based like it (internal version v becomes folder v-1, because mount calls the first version VER00000000).

Two passes:

1. projection, O(changes) and not O(versions x files): the live byte count is kept up to date while the history is replayed, so nothing has to be walked twice. It prints folders, entries, bytes and an estimate of the RAM (500 bytes per entry, which on the 1,349 version test said 15.15 GB against the 15.61 GB actually measured), then refuses if the bytes do not fit on the destination (bypass -space) or if the entries are more than ZIPALLMAXENTRIES (5,000,000, bypass -force);
2. build, with a progress line every 200 ms.

insert(), never operator[]: a default constructed DT calls franz_malloc() for its franz_block, and 32 million of those is not a good idea. DT has no destructor (it is commented out in the source, on purpose), so the copies share that pointer and nothing is ever freed twice.

-range cannot be allowed to filter the read, because the snapshot of version 300 is made of everything the 299 before it did. extract() puts it aside around read_archive() and hands it to the expansion, which uses it to choose which folders get written. Note that -range alone already forces all= 8, so zip -range 3 now writes the full snapshot of version 3 and not its delta.

The expansion runs right before the extraction marking loop, because from there on dt is walked to decide what to extract and iterators into it are taken (block[].files), so nothing may be moved around any more.

2. Windows filenames

franzrootlength() says how many leading characters are the "root" and must be left alone: 7 for //?/X:/, up to the share for //server/share/ and //?/UNC/server/share/, 3 for X:/, 2 for X:, 1 for a *nix absolute path.

franzsanitizepath() fixes everything after the root and nothing else:

• a doubled root (//?/z:///?/z:/something, which is what a long path stored inside the archive glued to a long path -to produces) is collapsed, up to 8 times;
• forbidden characters (< > " | ? * : and backslash) and control characters become _;
• with i_strict (that is -fixreserved) it also fixes what Windows accepts and then silently mangles: trailing dots and spaces, and the DOS device names.

Jidac::sanitizzawindows() rewrites the keys of dt with it, counts how many changed, and is called by extract() and extractw() unless -nosanitize. It never runs for zip, where those names are exactly what has to be preserved.

forcelongpath() is the other half: it turns any path into the extended //?/ form, expanding a relative one first, without caring about -longpath. Some operations (erasing, for one) never need compatibility with anything.

3. rd

erredbarras() was rewritten around three ideas.

erredbarrasfallito() is called on every failure: it counts it (g_rd_errors), remembers the first one and its object (g_rd_lasterror, g_rd_errorpath), and returns, so the walk goes on. The report at the end prints the count, the decoded Windows message of the first error and the object it was about.

Reparse points are checked before recursing: (attributi & FILE_ATTRIBUTE_REPARSE_POINT). A junction or a directory symlink is removed with RemoveDirectory on the link itself, which leaves the target alone. Without this, deleting a folder could erase files on the other side of the disk.

Names go through wtou() everywhere, instead of the old std::string(w.begin(), w.end()) that truncated every non ASCII character, and stermina() forces the extended syntax on the whole path before starting.

4. mount

The history of a file, instead of a copy per version

Both engines used to hold, per version, a map<string, record> of the whole tree. 65.2 replaces it with the changes:

:::cpp
struct JidacChange { unsigned version; const JidacRec* rec; }; // rec NULL = deleted
typedef map<string, vector<JidacChange> > JidacHistMap;

one entry per file, and inside it only the versions that really touched it. jidac_at(hist, v) is a binary search for the last change with version <= v, jidac_touch() appends one (twice in the same version is an overwrite). The records live in a deque, so pointers into it never move, and a file that does not change across 300 versions is stored once, with its fragment list stored once.

The core engine

Commented out, not deleted: from the Backend "core" (DISABLED) banner down to the end of CoreBackend, 469 lines are the original source with a // in front of each, verified identical to the original once the prefix is stripped. CoreFragLoc, CoreStrWriter and mount_tree_add()/mount_tree_finish() stay alive because jidac uses them.

mount_pick_backend() returns jidac for everything and keeps the original choice in a comment; open_backend() has the core branch commented out and says (available: jidac, auto); -backend core is refused as an unknown backend.

The ETA of the tree building

MountScanProgress could only count bytes of the archive, which is the right yardstick for the index scan and no yardstick at all for the phase that builds the directory trees. It now also counts units of work, one unit being one version's tree:

:::cpp
void begin_units(int64_t i_units);  // the phase counts versions, not bytes
void unit(int64_t i_done);          // i_done units are behind us
void subunit(double i_frac);        // how far the current one has got, 0..1

build_tree() feeds subunit(done/entries) while it walks, so a single version mount (the default) has an ETA too, and Jidac::mount() calls unit(v) before each prepare(v). Percentage and ETA are the same arithmetic as the byte based ones, and the line becomes 43.07% 00:00:40 eta 00:00:53 V00000581 sorting 72.871 files.

In -catpaqmode the phase used to emit nothing at all (it returned early on total_ <= 0); it now sends @SPK@EXT@ with percentage and ETA, and 0/0 in the two byte counters, because the GUI formats those with FormatFileSize() and a number of versions in there would read as bytes.

The totals walk that follows starts its own phase, so it does not inherit the units of the one before it.

5. Resources, zpaqfranz-full

resources.rc stopped at 11 while fai.bat was already compressing the MSI into 12.lib: windres ignored it, and the installer was not inside the executable at all. The proof is arithmetic, the sum of 1.lib to 11.lib is 4,388,866 against a resources.o of 4,389,656, which is 790 bytes of COFF overhead and no room for 1.28 MB of MSI. One line added to the .rc, and the object goes to 5,672,482.

The table in kickstart_resources() gained the entry:

:::cpp
elenco_risorse.push_back(risorse(12, "WINFSP", "winfsp-2.1.25156.msi", 2191360,
    "073A70E00F77423E34BED98B86E600DEF93393BA5822204FAC57A29324DB9F7A"));

and mysqldump.exe had its size corrected from 4,875,064 to 4,785,064 (the hash was right, the digits were transposed, and it only fed the progress bar).

estrairisorsa_in(number, destfile, size) extracts a resource anywhere, where estrairisorsa() always writes next to the .exe because that is where a DLL has to be. It returns false without a word when the resource is not there: that is a plain zpaqfranz.exe, not an error.

kickstart_mount() now looks for the MSI in this order:

	where	message
1	system temp folder	94204 reusing a previous attempt
2	zpaqfranz's own folder	94222 reusing
3	resource 12	94223 extracted from this very executable
4	download	94205, as before

Size and SHA-256 are checked whatever it came from, then the captcha and msiexec as before. The WINFSP package is left out of the kickstart_resources("") sweep on purpose: the MSI is not something zpaqfranz needs to run, and on a build without resources that sweep would mean 2 MB of download that nobody asked for.

6. Wildcards and patterns

expandwildcards(filename, recursive) is the *nix half of what the Windows shell does: opendir() on the folder part, ispath() on each name (case sensitive here), no recursion into a matching folder, and never expanding into something that is expandable again. myscandir() calls it when the argument holds a wildcard and the file does not exist as it is written.

matchesfilepattern(pattern, filename) is for dir: the pattern is applied to the name, not to the full path. It is deliberately not pushed into onlyfiles, which would match the whole path and fail unless the pattern started with *, and which the user may want for something else.

7. Franzen-only

franzenname(x) returns x when x is already a .franzen, and x + ".franzen" otherwise. is_file_frenzen() does not refuse an archive whose cleartext side does not exist, and OutputArchive never opens the cleartext file when the archive itself is the franzen one.

8. Fixes

fseeko() on Windows: SetFilePointer() returns INVALID_SET_FILE_POINTER both for a real error and for a perfectly good seek to offset 0xFFFFFFFF, so the only way to tell them apart is to call SetLastError(NO_ERROR) first and look at the error code only when the return value is that sentinel. Without it fseeko() reported whatever some unrelated API had left behind, a DeviceIoControl that failed long before being the usual suspect.

Jidac::mycopy(): edt is cumulative across scandir() calls, so a loop that walked it once per source counted and copied the same file as many times as there were sources. It is walked once now.

decodewinerror() takes an i_padded parameter: the 30 characters padding keeps the columns of the error report aligned, and a message printed on its own gets the whole text instead of a stump.

---

Note

Numbers in this document were measured on a single Windows 11 machine with mingw-w64 builds. The deflate ratios depend entirely on the data, the mount ETA depends on how many versions and how many files the archive holds.

Everything is built from one single .cpp, as usual:

g++ -O3 -std=gnu++14 -DZPAQMOUNT -I"C:/Program Files (x86)/WinFsp/inc" ^
    zpaqfranz.cpp -o zpaqfranz.exe -static -pthread ^
    -lws2_32 -ladvapi32 -luser32 -lurlmon -lole32

and zpaqfranz-full.exe is the same line plus c:\zpaqfranz\dll\resources.o.

Things to do

Testing on *nix A better help for new commands