This new build is of a new branch, includes several new features to be tested, and may therefore contain bugs.

In this case, please report, and I'll fix ASAP

-chunk

After (quite a lot) of digging here the fixed-chunk version, a long overdue improvement Of course there are a lot of complexity to be taken in account (aka: nothing is easy with zpaq)

zpaq does not allow to create archives of a fixed size: this implies that, generally, the first file is gigantic, and subsequent ones much smaller. This prevented splitting on optical media (e.g., Blue Ray), and this is bad (incidentally that's "where" I'm going to use it). The zpaq's operating logic doesn't really provide multiple output files, but now (maybe) it does 😄

Operation, which is still not fully integrated (for example, it is not supported in the backup command), is easy to activate. It is like a normal multipart archive, but with a switch that indicates the maximum size of the pieces Running with encrypted archives has been difficult, and it is still not 101% tested

zpaqfranz a z:\ronko_?? whatever-you-like -chunk 1G
zpaqfranz a z:\ronko_?? who-knows -chunk 500m

The -chunk gets straight number (2000000), K,M,G (100M), KB,MB,GB (500MB) The chunks are not guaranteed to be 100% exact, typically should be multiple of 64KB

The created multipart archive SHOULD be fully backward compatible, even with 7.15 (this takes a lot of efforts!)

the "real" problem is... Control-C

In other words, abrupt termination is not trivial to handle, because it is necessary to delete created but unfinished pieces, by identifying file handles in order to close them beforehand (otherwise the deletion will fail). Another problem to be solved is the a priori estimation of the number of parts needed, compared to the number of parts indicated (in other words, the number of ? in the file name). Next release

ADS filelists (alternate data stream)

Running on Windows' NTFS, for unencrypted archives, it is possible to store the file list in the Alternate Data Stream this way

zpaqfranz a z:\1.zpaq *.cpp -ads

Now the list command will take data from the ADS

zpaqfranz l z:\1.zpaq

To enforce "standard" way, use -ads in list

zpaqfranz l z:\1.zpaq -ads

Not very refined, a bit of test is necessary After (quite a lot) of digging I decided to use LZ4 to compress the ADS stream. It reduce the file size not much, but is extremely fast during the extraction phase and, even better, I was able to do it 'in chunks', making the size of the file list essentially limitless, "hacking" one of LZ4's examples. And yes, it is the first piece to have zpaq archives "mountable" as a filesystem (in the distant future)

As you know, or maybe not, when you ask the list of the files inside an archive every transactions (version) is readed from the start of the file

This can become slow over time (big archive with millions files, thousands versions)

TRANSLATION

Suppose you have File1 File2 File3 Inside version 1 And File4 File5 File6 File7 Inside version 2 (...) File7000000 File7000001 File7000002 Inside version 1000

If you ask the list first you have to decode version1, then version2 then version...1000

zpaqfranz already allow to store those lists inside the archive (-filelist) But if you ask for a list in the 1000th version, a lot of (slow) work is needed too

Manipulating ADS by new ads command

There is a new command to get ads (on Windows, of course)

Show    ADS filelist                    ads z:\1.zpaq
Rebuild ADS filelist                    ads z:\1.zpaq -force
Remove  ADS filelist                    ads z:\*.zpaq -kill

-fast: store inside zpaq archive a "kind of" file list (to be developed)

This is an experimental function for future use, which is extremely complex to make work properly Basically, it involves hacking the original zpaq format to hold additional data, while maintaining backward compatibility. It's a gigantic job, because zpaq 7.15 is definitely "choosy" about the format of the files it processes

For the 'curious' I give the recap The aim is to extract a file from a zpaq archive without having to read the zpaq file itself (i.e. without reconstructing its contents as zpaq normally does, from begin to end, skipping over d-ata blocks) For gigantic files this can be slow, even taking minutes. Using the 'trick' of storing the necessary information in the LAST stored file as a 'link', we essentially read the last 'dummy' file and from that go directly to the data Unfortunately, I was not able to eliminate the fragmentation altogether (without breaking backward compatibility), which would have made it possible to use LZ4 instead

On Windows it is much easier and faster to use ADSs (alternate data streams) but on *nix they obviously do not exist, so this is "why"_

zpaqfranz a z:\1.zpaq c:\dropbox -fast

When listing, if fast founded, decode / else enforce standard (opposite of -ads )

zpaqfranz l z:\1.zpaq
zpaqfranz l z:\1.zpaq -fast

As mentioned it is an area that currently contains essentially useless information

Getting password from console is changed, now should be run... everywhere

zpaq 7.15, when encounters an encrypted file and does not know the password (not specified with -key something) aborts. zpaqfranz instead asks the user from the keyboard, which is much safer (it doesn't end up in the command interpreter's history). This used to happen separately for each function (and was long-tested). Now instead it happens at the centralized level of the class that manipulates the incoming files. This is "risky" for new bugs, however it does make all commands inherit this feature (example dump)

Colors on Windows

There is now (limited) color support on Windows console, IF THE BACKGROUND IS BLACK (as it is known, or perhaps not, the console background historically can get 8 colors (a combination of single-bit R, G and B) with an additional level of intensity. But the new Windows' versions use 32-bit palettes, making it really hard to tell what the actual background color is. Windows for example "say" magenta, but... it is blue (!) That's why I decided to make it work only if black).

zpaqfranz also supports the NO_COLOR environment variable; if it is set (or the -nocolor switch is used) it will revert to normal mode

Colors on non-Windows? Maybe in the future. There are so many interoperability issues that I'm pretty skeptical about getting it to work on every system. We'll see

Smarter debug

There are now 4 switches for debugging (!) -debug, -debug2 and -debug3 show more information as they go. -debug4 activates a special mode, in which it writes debug files inside the z:\ drive (if any)

-longpath files

Support on Windows of paths longer than 255 characters is extremely complicated. There is no "serious" function in the Windows API to do something as trivial as locating the real name of a file. The "best" function, which works in almost every case, involves using COM, and I don't want to do that (too slow and too high a risk of leakage when used for thousands and thousands of calls, Scripting.FileSystemObject with GetAbsolutePathName) Now I take from KERNEL32.DLL getFinalPathNameByHandleW=(GetFinalPathNameByHandleW_t)GetProcAddress(h, "GetFinalPathNam...)

The -longpath switch in zpaqfranz is designed to operate on PATHS, not FILE At the request of a user, I have also included rudimentary support for individual files

Here the "spiegone" https://github.com/fcorbelli/zpaqfranz/issues/90

NEW BUILD = NEW BUGS

As usual, the more issue reports, the better the program becomes. The right place is https://github.com/fcorbelli/zpaqfranz/issues

Thank you for any help in improving the product

In extenso links https://encode.su/threads/4182-Color-or-not https://encode.su/threads/4178-hacking-zpaq-file-format-(!) https://encode.su/threads/4168-Virus-like-data-storage-(!)