zpaqfranz 65.8b

This is the release of the backup that looks after itself. It tells by e-mail how it went, it goes to the cloud and says OK or ERROR, it takes what can still be read of a dying disk. And the images got thinner, and found one more way out:

• The report by e-mail, sent by zpaqfranz: -mailfull and -mailprivacy at the end of any command, on every platform. Two logs: the one as it is for the owner of the data, the one without the names of the files for who looks after the backup. No external program any more.
• cloud: every phase is OK or ERROR, nothing in between; the upload goes on from where the remote archive ends; cloud ... c: -image sends the image of a drive, and asks for the administrator by itself.
• -rescue: the image of a dying disk. A read that fails is not insisted on: zpaqfranz jumps ahead and comes back, tens of failed reads instead of thousands.
• Thin images: the free clusters that came along with the used ones are zeros in the image. A volume with its free space full of old data: 53 MB instead of 4,978. The files are the same, the source is not touched.
• image -to x.vmdk: a virtual disk for VMware Workstation, VirtualBox, QEMU/Proxmox. Sparse (about as big as the used data), or flat with -raw.
• f: the live map for the fill of a folder too, zeros instead of random data, and f X: -test -force -zero -ntfs: zeros in the free clusters of an NTFS volume, the files untouched.
• Every version says what it is: who wrote it, on which system, how long it took, how much it read. i -verbose shows it, t checks that the versions are where they were written.
• Smaller: image out of Windows (the image of a /dev/sdX back to a raw file), kickstart, a restore of an image full of zeros eight times faster, the window that waits for an elevated run tells how it went.
• Fixes that matter: clusters over 64 KB, a raw image through VSS that came out short, password that wrote garbage with a wrong key, -to silently ignored with a wildcard, a help that showed a switch cut short.

This document has three parts:

• Part one: what is new, in short, for everybody
• Part two: the same things in more detail, for power users
• Part three: "lo spiegone", how it works inside, and why, for developers

---

Part one, what is new, for the user

1. The report by e-mail
2. cloud: OK, or ERROR
3. -rescue: the image of a dying disk
4. Thin images
5. <[inline_block>0](#5
6. f: the map, the zeros, the free clusters
7. Every version says what it is
8. The window that waits
9. image out of Windows, and <[inline_block>1](#9
10. Smaller things

1. The report by e-mail

zpaqfranz a z:\1.zpaq c:\data -stat -mailconfig c:\zpaqfranz\mail.conf
          -mailfull owner@x.com -mailprivacy monitor@provider.com -customer smith

At the end of any command (a, backup, cloud, t...) the log goes by e-mail. zpaqfranz sends it by itself (SMTP with TLS): -maila and mailsend.exe are gone, and it works on Windows, Linux, the BSDs, macOS, ESX and the NAS builds.

switch	meaning
-mailfull ADDR	the log as it is, the names of the files too: for the owner of the data
-mailprivacy ADDR	the log without the names of the files (totals, results): for who looks after the backup
-customer NAME	in the subject: OK, WARNING or ERROR, then the name, then FULL or PRIVACY
-mailconfig FILE	the ONE account that sends both (server, port, tls, user, password, from)

The typical use: one mailbox sends, two addresses get it. The customer reads everything; the provider sees that the backup ran, how much it took, how it ended, and not one name of a file.

The e-mail has a summary and the last lines in its body, and the whole log attached, zipped. An e-mail that cannot be sent is a WARNING (exit code 1) when the command itself was OK: the backup is good, but nobody knows.

2. cloud: OK, or ERROR

zpaqfranz cloud u:\prod.zpaq c:\server\*.* -host secure.example.com -user produser
          -ssh ~/.ssh/prod_key -remote /secure/backups/ -test -verify -key
zpaqfranz cloud u:\c.zpaq c: -image -host ... -remote /home/p/image/ -key

• If it ends OK, it is OK. Anything else is an error. The upload, the upload of the checksum, the final check and the deep check are OK or ERROR: a result that was a warning is now an error to be told.
• The upload goes on. A zpaq archive only grows at its end: zpaqfranz checks that what is on the server is the beginning of the local archive, and sends only what is missing. An upload that broke off is taken up again from there.
• A local folder is a remote folder, as an rsync would leave it: the final check compares the whole folder of the archive with -remote. A file here and not there is an error.
• cloud ... c: -image is a archive c: -image as it is (-raw, -novss, -rescue... too), then all the rest. From a normal prompt it asks for the administrator (UAC), the work goes on in a new elevated window, the first one waits and ends with its exit code.
• The password is asked twice for a new archive, here and in a and backup. An empty answer to -key is refused: the archive would not be encrypted.
• It does not wait forever any more. On ERROR, a wrong password, a refused captcha: what is on the screen stays for 60 seconds (a key ends the wait), then cloud ends. Before, it waited for a key: with nobody at the keyboard the scheduler never started the next run.
• cloud always lists the names of the changed files (as -stat does): for a log without them, -mailprivacy.

3. -rescue: the image of a dying disk

zpaqfranz a z:\dying.zpaq d: -rescue
zpaqfranz a /mnt/safe/sdb.zpaq /dev/sdb -rescue 64 -rescuetime 0

A read that fails costs seconds, because the disk insists by itself (7 to 60 seconds on a bad sector), and a dying disk may not have many reads left. The normal image loses nothing that can be read, but it pays one failed read for every bad sector: on a dead zone of 20 MB, more than 2,000 failed reads. Hours.

-rescue does it once, and takes all that reads at once:

1. forward, 1 MB at a time;
2. at the first read that fails, a jump ahead: 64 KB, then twice as far each time, up to N MB (-rescue N, default 16), until a read works;
3. from there back towards the trouble, up to the first read that fails.

Both edges of a bad zone are taken to the last good sector. What is in between is not tried: zeros in the image. About 15 failed reads for that zone of 20 MB.

The price: where the bad sectors are close to each other (less than 64 KB apart) the good ones between them are lost too.

-rescuetime S (default 2): a read that works but takes more than S seconds is a sick zone too. Its data is kept, then the jump out of the slow zone. 0: only the errors count.

-rescue implies -image. On Windows no shadow copy is made (nothing must be written on that disk).

4. Thin images

Nothing to ask for. An image of the used clusters is made of blocks of 2 MB, and a block is read whole when even one of its clusters is used: the free ones came along, with what is in them (deleted files, old data). On a volume where the free space is in small pieces that is every block, and what does not compress in the free clusters went into the archive.

Now the free clusters inside a block become zeros in the image. The source is not touched, the files in the image are the same.

a volume of 8 GB, 3.2 GB of files, the free space full of random data	archive
before	4,978 MB
now	53 MB

It happens when the volume cannot change while it is read: with a shadow copy (the default on NTFS), or with the volume locked. A volume read while in use (no VSS, not locked) is left as it is, with the warning of always. NTFS, FAT and exFAT. Volumes over 2 TB are imaged as before, without the zeroing: not supported (yet) there, and it says so.

The restore of such an image, full of zeros, was slow (the same was true of the image of a volume zeroed by hand): 66 seconds became 8 for a partition of 8 GB.

5. image -to x.vmdk

zpaqfranz image z:\c.zpaq c: -to d:\c.vmdk           (one sparse file)
zpaqfranz image z:\c.zpaq c: -to d:\c.vmdk -raw      (flat: c.vmdk and c-flat.vmdk)
zpaqfranz image z:\disk.zpaq 0: -to d:\disk.vmdk     (a whole disk)

A .vhd is what Windows mounts by itself. A .vmdk is what the hypervisors open: VMware Workstation, VirtualBox, QEMU and so Proxmox. 7-Zip opens it too, down to the files. Windows by itself does not.

	sparse (-to x.vmdk)	flat (-to x.vmdk -raw)
files	one	two: x.vmdk (a text of a few lines) and x-flat.vmdk (the disk, byte by byte)
size	about the used data: the zeros are not in it	the size of the disk
limit	2040 GB	none

From the images of the used clusters (NTFS, FAT, exFAT), from the raw image of a partition and from the image of a whole disk. No administrator needed.

A thin image shows here: the tortured test volume (8 GB, 3,056 MB used) is a .vmdk of 3,386 MB, written in 4.6 seconds. The .vhd of the same volume is about 8.1 GB.

To read the files. To start a virtual machine from it the image of the whole disk is needed, with its hidden partitions: that is for the next release.

6. f: the map, the zeros, the free clusters

zpaqfranz f z:\                               (fill the free space: the live map)
zpaqfranz f z:\ -zero                         (zeros instead of random data)
zpaqfranz f z:\ -nodashboard                  (the classic lines)
zpaqfranz f 3 -test -force -paranoid -zero    (zeros on ALL of disk 3: DESTROYS IT)
zpaqfranz f E: -test -force -zero -ntfs       (zeros in the free clusters of E:)

• f folder\ has the live map and the report of f -test: the free space is written and read back block by block, without the cache of the system. -nodashboard gives the lines of before.
• -zero with the triplet (-test -force -paranoid): zeros on the whole device, and zeros expected back.
• f X: -test -force -zero -ntfs: zeros in every free cluster of an NTFS volume, written straight on the volume. The files are not touched. It is what a thin virtual disk wants before being shrunk, and five seconds on the 8 GB test volume. The volume is locked meanwhile, so not the one of Windows and not one with open files: there, f X:\ -zero (files of zeros). -verify reads the free clusters back. Not NTFS: refused. Over 2 TB: refused, not supported (yet).
• The classic fill ended with exit code 0 whatever happened. Now a write that fails is an error (2).

7. Every version says what it is

zpaqfranz i z:\c.zpaq -verbose

VFILE-info: 2 of 2 version(s) say what they are
------------------------------------------------------------------------------------------
<  Ver  > zpaqfranz os        -m         time           bytes read    begins at byte  type
------------------------------------------------------------------------------------------
V00000001 65.8b     win64     84     00:04:50      221.620.522.452                 0  image of c:
V00000002 65.8b     win64     84     00:04:26      220.819.213.276    94.838.370.045  image of c:
------------------------------------------------------------------------------------------
                                     00:09:16      442.439.735.728

Together with the history of -fast (the default) every new version gets a small record: which zpaqfranz wrote it, on which system, the method, how long it took, files added, changed and removed, bytes read, and what it is (multipart, franzen, the image of a drive, tar, stdin). It costs nothing: no read more of the archive, a few bytes.

• i -verbose shows the table. When every version is an image the three columns of the files are left out (they would count the pieces of the image).
• t checks, for each version that has a record, that it is the version it was written as, that it begins where it began and with the fragment it began with. A version taken away from the middle, a piece of a multipart archive that is not the right one: an error, exit code 2.

8. The window that waits

C:\> zpaqfranz a z:\c.zpaq c: -image

Admin rights required => getting the power!
136.891s (00:02:16,536) (all OK)

a with -image or -vss, and q, from a normal prompt start themselves again in an elevated window. The first window waited, said nothing, and ended with exit code 2, always. Now it prints the last line every run has, the time of the whole thing and how the elevated run went, and it ends with its exit code: a script sees 0, 1 or 2.

9. image out of Windows, and kickstart

zpaqfranz image /backup/sda.zpaq /dev/sda -to /mnt/big/sda.raw
zpaqfranz kickstart -to z:\tools

• On Linux, the BSDs and macOS a -image is the device byte by byte. image now gives it back as a raw file (to be put back with dd, or looked into with losetup), there and on Windows too. Before, on *nix it printed the help and ended with exit code 0.
• kickstart (Windows, 64 bit) gets every external file zpaqfranz can use: the DLLs of ssh, curl and sodium, mysql.exe, mysqldump.exe, the WinFsp installer. The full build takes them out of itself, the others download them. The SHA-256 of every file is checked. Not in the open build.

10. Smaller things

• NTFS with clusters over 64 KB: the image of the used clusters did not understand them ("record MFT $Bitmap not good") and took the whole partition instead. A volume with clusters of 2 MB: a .vhd of 4.1 GB instead of 8.1.
• A raw image through a shadow copy (-raw -vss, and what -image falls back to) ended one cluster before the end of the partition: the .vhd made from it was not mounted by Windows. Now it is as long as the partition. The images of that kind already made stay short: they go back on a partition, not to a .vhd.
• -image -novss on a volume just written said "read while in use" and ended with exit code 1 with nobody using it: the lock was tried once. Now for three seconds.
• password with a wrong -key wrote an archive of garbage, and with -force it wrote it over a good one. Now the key of the source is checked first: nothing written. -key2 . takes the password away without a terminal (it encrypted with the key ".").
• a "folder\*" -to x: -to was not applied and nothing was said. It is refused now, as x does since 65.6.
• x over a file left half way by a killed extraction: when it had already its final size it was skipped as good. Now, when its date is not the archived one, its content is compared: a warning, exit code 1, and the file is not touched (-force to write it again).
• The help cut its headers: twelve lines came out short or glued to the text, -test -force -paranoid as -test -force -paran. A help that shows a wrong switch is a bug.
• An upload taken up again that sent only a part, when what was missing on the server was more than what was already there (an archive that more than doubles, a first upload broken before its half). The check found it, and a second run fixed it: now the first one is enough.
• mount x.vhd from a normal prompt: Ctrl+C in the window that waits did not detach the disk, the elevated window stayed there. Now the elevated one follows the first: gone the first, detached the disk.
• mount archive -test said FAILED with paths over 260 characters.
• -always took the folders too: a tried to open them as files.
• The first line of _crc32.txt has the name of the archive only, without its path: that file goes to the cloud as it is.
• "Destination too small" suggested -space, that does nothing there.
• On Linux, after a read error of a device, good data around the bad sector was lost (see part three).

---

Part two, the details, for power users

1. The e-mail: who sends, what is taken away
2. cloud, phase by phase
3. -rescue: what is lost, and what is not
4. The thin image: when, and what it says
5. image: every combination, again
6. f: every way
7. t and the records of the versions
8. What can change for a script

1. The e-mail: who sends, what is taken away

who sends	how
ONE account, for both e-mails	-mailconfig FILE: a text file of key = value (server, port, tls, user, password, from)
the same, without a file	-mailserver -mailuser -mailpassword -mailfrom (-mailport, -mailssl...: see h work)
a second account for the log without names (a special case)	-mailprovider FILE, a file as above with to too; or zpaqfranz-mail.conf next to the executable

mail.conf:   server = mail.provider.com
             port = 587
             tls = starttls
             user = log@provider.com
             password = pw

-mailcafile FILE	the CA certificates (.pem), where the system has none (ESX, some NAS)
-mailinsecure	no check of the server
-mailtimeout, -maillog	as they say
-verbose, -debug	the log of the sending, the SMTP dialogue

What -mailprivacy takes away: the names of files and folders become ***, the |STAT| lines and the listings are left out. Totals, times, versions, results and errors stay. The purge is best effort, and never with -debug (those lines can hold names).

The progress lines are not in the report: what is redrawn in place on the screen (percentages, the lines of the upload) does not go in the e-mail.

The log attached is zpaqfranz_log.zip (or zpaqfranz_log_privacy.zip); over 15 MB of zip it is not attached. Who starts an elevated window does not send: the elevated run does.

Sent, and accepted by a real server, from: Windows, Fedora, FreeBSD 14.2, OpenBSD 7.9, macOS 12.7, ESX (a static build for a gcc 3.4.6 system), the static NAS builds for x86_64 and, under qemu, ARMv5, ARMv7, ARMv8. -DNOEMAIL builds without it.

2. cloud, phase by phase

phase	what	when it is ERROR
archiving	a, with -stat always on (-turbo as in a); with -image, a archive X: -image	as a
-test	the archive is tested	as t
versions	the list of the versions	-
-verify	the archive read again, its CRC-32 against _crc32.txt	not the same
upload	SFTP, in append: the remote must be the beginning of the local one, then only the tail	not a beginning (87455), or the transfer fails
quick check	size, and three samples of 16 KB: start, middle, end	not the same
checksum	_crc32.txt uploaded (always whole)	not uploaded when the archive did not get there (91555)
final check	the whole local folder of the archive against -remote	a file here and not there, or different
deep check	-md5deep, -sha1deep, -sha256deep: the hash of the remote file, computed there	not the same; not done when an upload failed (91556)

• The deep check runs once, at the end: it ran inside both uploads, and the final one never ran.
• -onlyupload: the summary shows only what was done.
• -force overwrites the remote archive: it asks the captcha.
• A new archive: the password twice (51852 if they differ, and no e-mail: somebody is at the keyboard). -key with an empty answer: 51853.
• Still a warning, yellow: a folder to save that is not there, an e-mail not sent.
• Not an administrator after asking for it (UAC off, no desktop: ssh, a scheduled task): 91553, said from inside cloud, so that the e-mail of the error goes. Run it elevated (a scheduled task: with the highest privileges).
• The shadow copy of a cloud -image is released right after the image, not at the end of the program: the upload of a C: takes hours.

3. -rescue: what is lost, and what is not

Measured on Fedora with real failures (device-mapper over 256 MB: one bad sector alone; 40 stretches of 4 KB every 256 KB; one bad sector every 16 KB for 10 MB; a dead zone of 40 MB; a slow zone of 10 MB, 3 seconds a read; the last 64 KB), sector by sector against the source:

	failed reads	good sectors lost
without -rescue	3,987	0
-rescue	195	9.9 MB (the zone of one bad sector every 16 KB) + 9.2 MB (the slow zone)
-rescue -rescuetime 0	195	9.9 MB

Mind what is counted: device-mapper fails at once, a real disk does not. At 7 seconds each, 3,987 failed reads are almost 8 hours, 195 are 23 minutes. No sector was ever wrong: lost means zeros.

-rescue N	the longest jump, in MB (default 16). Bigger: fewer reads inside a long dead zone
-rescuetime S	a slow read is a symptom (default 2 s). It is not a timeout: zpaqfranz cannot stop the disk while it insists on a sector
the disk can be told to give up sooner	smartctl -l scterc,20,20 /dev/sdX (2 s); on Linux also /sys/block/sdX/device/timeout
Windows	no VSS (73909): nothing must be written on that disk
*nix	the device is read without the cache of the system

There is no map of the bad sectors in the archive, and no second pass: once, what reads at once. At the end the image says how many reads failed, how many jumps, how much is zeros.

4. The thin image: when, and what it says

the volume	the free clusters in the image
with a shadow copy (the default on NTFS)	zeros
locked (-novss, FAT, exFAT, no VSS possible)	zeros
read while in use (no VSS, not locked): warning, exit code 1	as they are
over 2 TB	as they are, and 45762 says so
-raw	everything, byte by byte: a raw image is not thin

With -verbose:

free clusters inside the blocks read: 4.81 GB as zeros in the image

• The list of the used clusters is the one of the shadow copy, not of the volume as it is while it is read: files deleted (2.6 GB) and written (240 MB) during the image are in the image as they were.
• A block with no used cluster was never read and never stored: nothing changes there. What changes is inside the blocks that are read.
• With clusters of 2 MB a block is a cluster: nothing to zero.
• An archive begun by 65.7 and continued by 65.8 is fine: the old versions are what they were.
• The other way to the same result, for a volume that is already a virtual disk: f X: -test -force -zero -ntfs, that writes the zeros on the volume itself.

The restore, the same image of 8 GB (53 MB of archive), on the test VM:

	before	now
on a partition	66.3 s	8.2 s
on a partition, -raw	65.4 s	8.1 s
to a raw file	65 s	8.7 s
to a .vhd	13.5 s	13.5 s

The same bytes, faster.

5. image: every combination, again

the archive has	-to x.vhd	-to x.vmdk	-to x.vmdk -raw	-to x.raw	-to G: -image
used clusters (NTFS, FAT, exFAT)	dynamic .vhd	sparse, our MBR, the partition at 1 MiB	the same disk, flat	the partition	the used blocks (-raw: every sector)
raw partition	dynamic .vhd	sparse, our MBR, the partition at 1 MiB	flat	the partition	every sector
whole disk (3:)	the disk, as it is	the disk, sparse	the disk, flat	the disk	refused (09411)
a *nix device (/dev/sda)	refused (62307)	refused (62307)	refused (62307)	the device	refused (62308)

• A .vmdk has sectors of 512 bytes (62309 for a source that has not).
• A sparse .vmdk over 2040 GB: not supported (yet), 62310. The flat one is the way.
• The flat .vmdk of a whole disk is the disk: the same bytes, the same hash.
• A file already there is not overwritten without -force (62311): x.vmdk, and x-flat.vmdk.
• A disk full while writing: an error (2); what was written is left there.
• A .vmdk is a disk, not a partition: the image of a letter gets the same MBR of ours that the .vhd has.
• On a volume with its free space in small pieces a sparse .vmdk is much smaller than a .vhd: its grains are of 64 KB, the blocks of a .vhd of 2 MB, and one used cluster keeps a whole block.

6. f: every way

command	what it does	writes on
f X:\	fills the free space with files of data never repeated, reads them back; live map, report, verdict	files in ztempdir, deleted at the end
f X:\ -zero	the same with zeros	files
f X:\ -nodashboard	the classic lines, a chunk of 512 MB at a time	files
f X: -test	reads the whole device: READ ONLY	nothing
f X: -test -force -paranoid	the triplet: writes every block, reads it back. Destroys everything	the device
... -zero	the triplet with zeros	the device
f X: -test -force -zero -ntfs	zeros in the free clusters of an NTFS volume	the free clusters only

About -zero -ntfs:

• It asks a captcha. It warns that the shadow copies of that volume (restore points, previous versions) can be lost: seen, one was gone after the zeroing.
• The volume cannot be locked (68779): nothing written, exit code 2. No fall back, no forced dismount.
• A cluster is written only when it is free twice: for NTFS before the lock, and in the $Bitmap on the disk after it.
• No read back unless -verify.

In the fill of a folder the speed along the space says little (the files are where there is room): jumps between zones are shown, and do not make the verdict. Stalls, slow blocks confirmed, errors and wrong data count as in -test.

7. t and the records of the versions

The record is written with the history of -fast: not with -nofast, -index, -chunk, -append, -715, nor when the archive goes to stdout. A version that adds nothing gets no record (and is not written at all, as before).

t says	meaning
65428! version N was written as version M	versions are missing, or out of place
65429! version N begins at byte X, it was written at Y	the archive before it is not the same
65434! version N begins with fragment X, it was Y	fragments are missing (or too many) before it

What it cannot say: a version without a record (an older zpaqfranz, zpaq 7.15), a tail that is not there any more: an archive cut after a version is an older, valid archive. For that, something outside is needed (_crc32.txt, the size in the e-mail).

zpaq 7.15 and the older zpaqfranz read these archives as before. The older zpaqfranz show the record as one more deleted entry in l -all.

8. What can change for a script

before	now
-maila, mailsend.exe	-mailfull, -mailprivacy: sent by zpaqfranz (97840 if -maila is still there)
cloud: a phase ending with a warning, exit code 1	ERROR, exit code 2
cloud on error: waits for a key	60 seconds, then it ends
cloud, a, backup with -key on a new archive: asked once	twice
a ... -image (-vss, q) not elevated: exit code 2 always	the exit code of the elevated run, and a last line
the image of NTFS, FAT, exFAT: the free clusters as they are	zeros (a smaller archive, the same files)
-image -novss right after the volume was written: exit code 1	the lock is tried for 3 seconds
a -raw -vss image: one cluster short	as long as the partition
image arc /dev/sda -to x.raw on *nix: the help, exit code 0	the raw file, or an error (2)
image ... -to x.vmdk: a raw file with that name	a .vmdk
a "dir\*" -to x: -to ignored, exit code 0	refused (71410), exit code 2
password -key2 .: encrypted with the key "."	the password is taken away (54641)
password with a wrong -key: garbage written	nothing written (54642)
x over a half written file of the right size: skipped, 0	a warning, 1
f folder: exit code 0 whatever happened	2 on a write that fails
f folder: the classic lines	the live map (-nodashboard for the lines)
f X: -test -force -zero -ntfs, image -to x.vmdk over 2 TB	refused: not supported (yet)
t: versions moved or taken away, not seen	an error (2), when they have their record
i: the list of the versions	the same; with -verbose the table of the records too
_crc32.txt, first line: the archive with its path	the name only
-always on a folder: an error of a	the folders are not taken

---

Part three, lo spiegone, for developers

1. The capture, and the purge
2. The upload in append
3. -rescue inside, and the cache of Linux
4. VFILE-info: a protocol
5. franzusb on files, and a locked NTFS that does not answer
6. The thin image
7. The restore that was slow
8. <[inline_block>0](#8
9. Asking for the administrator
10. How it was tested
11. Fixes
12. Tried and thrown away
13. Known limits and open points

1. The capture, and the purge

Everything printed goes through one place, and from the very first line (mailreport_avvia, before the parameters are read) it is kept in memory twice: as it is, and purged. The purge is made at the source: a name of a file is printed with %Z (the format of zpaqfranz for a path), and in the purged copy a %Z is ***. printUTF8, list_out and the |STAT| lines know they are names too. A last net looks for what still seems a path.

So the rule for new code: a message that prints a name uses %Z, never %s, or the name ends up in the log of who must not see it.

The progress: a print that ends with \r takes back the line it began, "as on the screen" (in about a hundred places the text and its \r are two prints: the texts stayed, all on one line). A display redrawn in place by a thread (the lines of the upload) is between mailcattura_live(true) and (false): what that thread prints does not go in the report, the errors of the others do.

mailreport() sends: called by main, by seppuku() (a command that dies must tell too) and by cloud() before its banner. The library that talks SMTP and TLS and the one that makes the zip are in the source, between their own markers (they are generated: not to be touched by hand).

2. The upload in append

A zpaq archive is written forward: a new version is appended. So the remote file, when it is good, is a prefix of the local one. The check is "quick": the size, and the hash of samples of both; then only the tail goes up, and a second quick check (size, three samples of 16 KB: start, middle, end) says how it went. The deep check (-sha256deep and the others) asks the server for the hash of the whole file.

The bug of the resume: the local file was already positioned at the point to start from, and CURLOPT_INFILESIZE_LARGE was the size of the tail, with CURLOPT_RESUME_FROM_LARGE the point. But libcurl takes the resume point away from the file size by itself: it sent (tail minus resume) bytes. When the tail was smaller than what was already there the count went below zero, which for libcurl is "size unknown, up to the end": that is why it usually worked. When the archive more than doubled in one run, or a first upload was broken before its half, the remote stayed short. A valid prefix, so the next run finished it. The size given is the whole local file now.

The quick hash of a file shorter than 64 KB read all of it instead of its first N bytes: the append of a tiny archive was always refused.

3. -rescue inside, and the cache of Linux

img_rescue*, after img_errore(). A state machine over the reads of an image, the same for the three readers (the used clusters, the raw partition, a *nix device): forward at 1 MB; a failed read is found down to 64 KB and then to the sector; the jump doubles from 64 KB to the limit; the landing is halved until it stands on good data; then back by 64 KB and by sectors to the last good one. A slow read inside a jump born of a real error counts as a good one, or good islands would be jumped over.

On *nix the device is opened without the cache (O_DIRECT, F_NOCACHE), every read is of whole sectors, and a descriptor that answers EINVAL (a file on a filesystem of 4 KB sectors) is opened again with the cache.

The cache of Linux loses good data. Found while testing, and it is of the normal image too, not of -rescue: after a sequential read, one bad sector makes the kernel fail 512 KB on that descriptor (the read-ahead); with a new descriptor, the page of 4 KB; with O_DIRECT, the sector. And after an EIO it reads one page at a time. On a device with many failures: 5,503 good sectors lost (2.75 MB), 9,497 failed reads. Now what is read again after an error goes through a second descriptor without the cache: 0 good sectors lost, 3,987 failed reads.

Failures made up by a hidden switch (-rescuefake) are above the kernel and do not see any of this: every change to the reads of an image on *nix must be tried on a device that really fails (device-mapper, error and delay).

4. VFILE-info: a protocol

One more entry with date 0 in the last index block of the version, next to the pointer of the -fast history, in clear:

VFILE-info:1|v=2|ps=94838370045|fr=1234567|iu=4|id=1|zv=65.8b|os=win64|m=84|im=c:|du=266000|ba=220819213276|fa=4|fu=0|fd=0

v the version, ps where it begins, fr its first fragment, iu and id the records of its index (with a date, and deletions); then what it is and what it did. Keys only when true for mp (multipart), fz (franzen), im (image, and of what), tar, si (stdin).

It is a protocol, to be extended: key=value, a key that is not known is ignored, a key that is not there means that check is not made, a record of a higher generation (the number after the colon) is ignored whole. Keys are only added, a meaning never changes.

What it knows before writing is taken before writing (vinfo_prima), the rest when the index is made (vinfo_riga): nothing is read from the disk for it. A first version of this (-112, see section 12) kept the CRC-32 of the archive too, and to know the one "before" it read the archive again. Thrown away: no metadata must cost one more read.

The pointer line of -fast (ZPFL1|...) was not made longer: its reader wants exactly its eight fields, and every zpaqfranz from 65.4 would fall back to the slow listing.

5. franzusb on files, and a locked NTFS that does not answer

The fill of a folder is franzusbtest on a "device made of files": franzusb::apricartella, files zchunk_NNNNN of 512 MB written without the cache (FILE_FLAG_NO_BUFFERING; on *nix O_DIRECT when the filesystem has it, else fsync and posix_fadvise). The same passes, map, report and verdict of f -test. -zero is in the generator (zeri()): zeros written and zeros expected, for the files and for the triplet.

franzzerontfs writes the free clusters of a volume raw. The order is what took the time:

• FlushFileBuffers on the volume first: NTFS keeps for a while the clusters of what was just deleted, and they would look used.
• The map of the used clusters as NTFS says (FSCTL_GET_VOLUME_BITMAP), before the lock: a locked NTFS volume does not answer any more (error 21: the lock flushes everything and lets the volume go).
• FSCTL_LOCK_VOLUME, a few tries.
• The $Bitmap as it is on the disk, after the lock: boot sector, record 6 of the MFT, its fixups, the runs of its unnamed $DATA. What was allocated between the first map and the lock is there.
• A cluster is written only if it is free in both. Anything not understood on the disk (a $Bitmap spread over more MFT records): nothing is written.

6. The thin image

franzimager::azzeraliberi(): after a block is read, and after the clusters of the files left out are cleared, the runs of free clusters inside it are set to zero in the buffer. Only when m_usingvss || m_bloccato: the list of the used clusters must be the one of the moment the data is read. With VSS both come from the same device, the shadow copy.

The last cluster. segnaultima() says the last cluster of the volume is used, always: its block must be in the image for what comes after it (the copy of the boot sector of NTFS, in the last sector of the partition). When that cluster was free it stayed in the image with its old data, the only free cluster not zeroed. It was found by the comparison byte by byte, on a volume extended after it was filled. m_ultimolibero remembers how it was.

Clusters over 64 KB: in the boot sector of NTFS the byte "sectors per cluster" above 0x80 is an exponent (2 to the power of 256 minus the byte). It was read as a number: 2 MB clusters were 244 sectors.

The raw image through VSS: the device of a shadow copy ends where the volume ends, one cluster (or less) before the end of the partition. The tail is now read from the physical disk (42287).

The lock: FSCTL_LOCK_VOLUME is refused for a moment on a volume just written, or looked at by the antivirus or the indexer. Ten tries, 300 ms apart, only while the answer is "access denied".

7. The restore that was slow

Jidac::extractstdout is the engine that hands the fragments over in order, from a cache of decompressed blocks, with worker threads that decompress ahead. For every fragment it looked ahead for the next blocks to ask for, starting again from the current fragment, and when everything ahead was already cached or asked for it walked to the end of the file without finding anything to do. An image full of zeros is a long row of the same fragment (about 50 KB each, all in the same block): the time grew with the square of their number.

Now a cursor (scan_voce, scan_it, scan_idx) remembers where the last scan stopped, and the next one goes on from there; it is cleared when blocks are thrown out of the cache (the eviction, the reset after a stall), because then what is behind it can be needed again. The same output, byte by byte, on the other roads through that engine too (x -recover, pp, -stdout of a file not stored in order, zip -deflate).

8. franzvmdk

A writer of the two kinds of hosted .vmdk that everybody reads.

Sparse (monolithicSparse): a header of one sector (KDMV, version 1), the text that describes the disk inside the file (20 sectors), the grain directory and the grain tables twice (the "redundant" copy first: VMware and QEMU write both, and so does zpaqfranz), then the grains of 64 KB one after the other as they come. A grain table says where each of its 512 grains is in the file, in sectors, in 32 bits: so the file cannot be longer than 2 TB. A grain of zeros is not written: its entry is 0, and who reads gets zeros. No checksums anywhere. The capacity is known before the first byte, so the tables have their place from the start and are written at the end.

Flat (monolithicFlat): the disk byte by byte in x-flat.vmdk, and x.vmdk with the text:

# Disk DescriptorFile
version=1
CID=f539468c
parentCID=ffffffff
createType="monolithicFlat"

# Extent description
RW 16744448 FLAT "x-flat.vmdk" 0

# The Disk Data Base
#DDB

ddb.encoding = "UTF-8"
ddb.virtualHWVersion = "4"
ddb.geometry.cylinders = "1042"
ddb.geometry.heads = "255"
ddb.geometry.sectors = "63"
ddb.adapterType = "lsilogic"

ddb.encoding: without it VMware reads the name of the flat file in the code page of the PC, and does not find a name with an accent.

The bytes come from where the .vhd takes them. The image of the used clusters is a row of records, a sector of bitmap and a 2 MB block of the virtual disk (our MBR and the partition at 1 MiB are already in it), and the blockmap says which block each one is: preparavmdkthin and its handler put each block in its place. A raw image goes down the road of the raw image to a .vhd (preparavhdraw: the MBR for a partition, a whole disk as it is), told to hand its blocks to franzvmdk instead (setvmdk).

One thing to know: franzimager is copied with the Jidac that holds it (Jidac jidac(*this)), so the writer is a pointer there, not a member.

9. Asking for the administrator

Three places ask, in two ways.

cloud -image and mount x.vhd: ShellExecuteExW with runas and the command line as it was typed (GetCommandLineW: the quotes of a path with spaces, or of a password, are still there), a hidden -elevated added (never twice), the handle waited for, its exit code taken. For cloud it is done before anything is read from the keyboard, or the password would be asked twice.

a -image, a -vss, q: the older way, through the shell. It now hands the exit code of the elevated run over as 100 plus the code (a code of the shell itself is not taken for it), 222 when nothing started, 223 when the code cannot be read; and -elevated is added there too. ultimariga_tempo() is the first half of the last line of every run, taken out of main to be printed by the window that waits too.

mount x.vhd: the elevated window gets -elevatedpid, the process that asked for it. It opens it (SYNCHRONIZE) and looks at it in its loop: gone the first window, it detaches and ends.

-elevatefake (hidden) starts again without runas, in a hidden window: on a test bench there is no UAC question to answer.

10. How it was tested

The images, on a virtual machine with a second disk of 8 GB to destroy at will. tortura.ps1 makes NTFS as complicated as it can be: every kind of file (sparse, compressed, encrypted, 4,000 extents, 1,023 hard links, 300 streams, reparse points, a path of 25,000 characters...), and holes: the volume is filled, two files out of three are deleted, so that the free space is in about 17,000 pieces. Then, with tools that are not zpaqfranz:

• the manifest (manifesto.ps1): one line for each entry of the volume, with the raw Win32 calls: attributes, sizes, dates, links, file id, short name, security descriptor, reparse data, every stream with its SHA-256. After a restore it must be the same in everything;
• chkdsk, and the image mounted by Windows (a .vhd) or by OSFMount (a .vmdk);
• the imprint (t_esatto.ps1): for every cluster, used or free (as Windows says) and a hash of its bytes. The disk is made read only first, so NTFS mounts it read only and not a byte changes between the imprint of the source and the image. Then: every used cluster the same, every free cluster zeros, the same bitmap, the same bytes after the last cluster.

thin image, the whole battery: clusters of 4 KB, 64 KB, 2 MB, MFT records of 4 KB, a volume extended, FAT32, exFAT	266 checks; then 173 with the last build
to a .vhd and to a raw file	exact, byte by byte
on a partition	7 to 32 clusters differ, all of NTFS itself ($LogFile, $Mft, $TxfLog, $UsnJrnl, the index of the root): Windows writes them mounting the volume
.vmdk, sparse and flat, from thin and raw images, a whole disk	64 checks: exact through OSFMount; the flat one of a disk has the hash of the disk
.vmdk read by others	vmware-vdiskmanager: consistent, and its conversion of the sparse one to a full disk has the SHA-256 of the flat one written by zpaqfranz; qemu-img: check without errors, the same SHA-256; 7-Zip: 17,313 files inside
f, -zero -ntfs	free clusters not zeros: from 1.26 million to 0, the manifest the same

The rest: cloud against a real SFTP server (archives that double, tails of a few bytes, uploads cut and taken up again, a remote file changed); the e-mails to a real server from every platform, and to a fake one that keeps what it gets, to read what really goes out; -rescue on devices that really fail (device-mapper) and with made up failures on Windows and on every *nix; the password, the wildcards, the half written files, with scripts that compare the old build and the new one.

Built on Windows (g++ 14.2 ucrt64: plain and with the mount; the SFTP, open, 32 bit and old compiler variants checked for syntax) and Fedora 44 (gcc 16.2); the part of the series up to the e-mail and -rescue also on FreeBSD, OpenBSD, macOS, ESX and the NAS targets, and there with the full and the open build of Windows too. The autotest is all OK on Windows and Fedora. The source is 234,596 lines (8.5 MB).

11. Fixes

• password: -key2 . went through the hashing of every key, so the branch "enter . for no password" was never reached. Compared as a hash now. The key of the source is checked as checkpassword does before anything is written (and before -force deletes); the copy and the final check are in a try: on any failure the output is deleted.
• a with a wildcard and -to: rename() replaces a prefix, and dir/* is a prefix of nothing. Refused in testparametriadd() (71410), the same choice made for x in 65.6: rename() is used by dozens of commands. A file that is really called q?.txt (on *nix it can be) is not a wildcard.
• x, "exists, skip": with more threads the blocks of a file are written out of order, and a killed x can leave a file with its final size and holes inside. The date is the hint (the right one is set only when a file is complete), the content is the verdict (equal(), the SHA-1 of the fragments).
• image on *nix: the parser knew the command on every platform, the dispatch was under #ifdef _WIN32: the help, and exit code 0. restoreimagedevice() is outside the ifdefs.
• kickstart: the list of the resources (name, size, SHA-256) is one function now, used by the command and by who extracts them at need.
• The help: scrivi() cut a header longer than its column. A long one is printed whole on its own line, the description below.
• -always marked the folders too: skipped.
• mount -test on Windows did not use \\?\ over 248 characters.
• f, the classic fill: writes and closes are checked, exit(0) taken away, the 99% is really written (the last chunk can be partial), the cache is flushed and dropped before the verify, the hash of the zeros is computed once.
• The message of image "destination too small" suggested -space, which is not read there.
• cloud: the flags of the deep check were "saved" and the copies cleared, not the flags; with -turbo the files went through the plain add(); the summary of -onlyupload showed as OK phases never run.

12. Tried and thrown away

-112: the CRC-32 of the archive inside the archive ("the European number for emergencies"). A record with the size and the CRC-32 of the archive before and after each version, and a t that read the whole file in pages and checked them. It worked, on multipart and encrypted archives too. But to know the CRC "before" on an archive that had no record yet it read the archive again, and a metadata that costs a read of a 200 GB file is not a metadata. What is left is VFILE-info: only what is known without reading anything.

A map of the bad sectors in the archive of a -rescue, and a second pass to try them again: no. The copy of a dying disk is made once.

Two SMTP accounts as the normal case, one of the customer and one of the provider: the normal case is one mailbox and two addresses. -mailprovider is still there for who wants the second account.

Hiding the password of the mailbox in the configuration file: it would be hidden from nobody who wants to read it. In clear, and said.

13. Known limits and open points

• Not tested: the elevated window with a real UAC question. I do not have UAC at all 😄 . Volumes over 2 TB: there the free clusters are not zeroed and f -zero -ntfs refuses. Disks with sectors of 4,096 bytes, clusters under 4 KB. A flat .vmdk over 2 TB. The changes after -rescue (the thin image, f, the .vmdk) on the platforms that are not Windows and Fedora: they are Windows code, the others compile them away.
• A .vmdk was read, not run: by the disk tools of VMware and QEMU, by OSFMount, by 7-Zip. Not attached to a virtual machine that was then started. ESXi wants its own kinds of .vmdk: not tried there.
• To start a virtual machine from an image the whole disk is needed, with the EFI partition and the others: a archive 0: -image. The image of C: alone does not boot, whatever the format.
• VHDX is still not written. It would serve for more than 2040 GB, for sectors of 4,096 bytes, and to boot in a second generation Hyper-V machine. The flat .vmdk has no limit of size. I do not use Hyper-Microsoft VM, therefore not very interested in VHDX.
• The image of a *nix device goes to a raw file only: not to a .vhd, nor to a .vmdk.
• -zero -ntfs: the bitmap is in memory (a bit a cluster); a $Bitmap spread over more MFT records is refused.
• t exits with 1, and says "VERDICT: OK", on an archive with one byte changed in the middle of a block (the files that use it are told corrupted). A fast l does not tell a missing piece in the middle of a multipart archive (l -nofast and t do).
• x -recover with a cache smaller than three blocks (-ramsize 40MB) throws blocks away and decompresses them again without end. With the default cache (1 GB) it does not happen.
• An image -image -novss taken a few seconds after a chkdsk or a resize of the partition can find the volume busy for more than the three seconds of the lock: the warning, and exit code 1.
• A shadow copy already on a volume can be deleted by Windows during a -zero -ntfs (its storage cannot grow while the volume is locked).
• backup asks the password twice at every run (it looks for an archive that, in a multipart, has another name).
• The limits of 65.7 that are still there: parallel reads of an image, a .vhd whose extraction is killed is left sparse, the files left out with -not are in the .vhd with zeros inside.

And finally... the ... allin!

This is an example of how to create a local, encrypted backup, send it to a remote SFTP server, check it, and send two log messages: one that has been purged (for privacy) and one that is complete (with details of the files added, modified, and deleted).

zpaqfranz.exe cloud franco_whatever.zpaq c:\zpaqfranz c:\wallpaper -not musicall -key lapasswordona -remote /home/franco/repository -host sftp.somewhere.com -user thegoduser -port 23 -ssh keyfile_toload -stat -test -verify -ignore -mailserver mail.yourisp.com -mailport 587 -mailuser provona@yourisp.com -mailpassword "thepassword" -mailfrom provona@yourisp.com -mailfull iamtheclient@whoknows.com -mailprivacy iamtheprovider@power.com -customer franco_backuppone