Michael Lackner

Hello gianluca, Yes, of course, I've attached it for you. Note that in the meantime I found out a few things though: The parallel load will not go up immediately, it'll take some time for it to do so. Also, from time to time, load will drop to just a few cores for a while, only to ramp it back up later on. So if you do see unexpectedly low loads across your cores with those settings, give it an hour or so, then check again.

Here's an update. The difference is mind-blowing! I didn't think it would have such a massive effect. I've attached a screenshot showing off the current core load & speed! It's sitting at 33 - 34 MiB/s with the maximum solid block size of 64GB. 24 physical cores are well-loaded here. I guess this setting will reduce decompression speed for individual files if I understand it correctly, but as said, that's fine. Thank you!

Currently, I'm using the GUI, but CLI is also fine. In my experience, smaller dictionaries result in sometimes significantly larger archives when compressing large data sets. So I want the largest dictionary possible to reach the smallest possible archive size. This is the main goal. Making things more parallel / faster is the secondary goal. I've never even considered changing the solid block size (just read documentation about how this works). I'll try enlarging it, thanks!

Sam, I just compiled lzip, lzlib and tarlz on my Linux box to play around with it a bit, but the biggest problem with this is that 7-Zip can't unpack its archives on Windows. It seemingly has no decompressor for lzip. While I do have several CygWin installations set up already, I'm not going to compile+link a tarlz redistributable for people to use for this, as we agreed on 7-Zip for decompression (because most people have it installed on Windows). So my question regarding any possibilities for parallelizing...

Thank you for your reply! One reason why I wanted to keep it in one archive and not split it is that by keeping the data set large, the payoff of using block-level deduplication would be greater, especially with a monstrous dictionary / dedup hash table in RAM. I'm assuming (based on my previous tests with gzip, bzip2 and LZMA2 in xz and 7z) that LZMA2 is simply, generally "the best" for compressing rather "noisy" data. Like binary programs and stuff. Naturally, I could look at the data and identify...

Thank you for your reply! One reason why I wanted to keep it in one archive and not split it is that by keeping the data set large, the payoff of using block-level deduplication would be greater, especially with a monstrous dictionary / dedup hash table in RAM. I'm assuming (based on my previous tests with gzip, bzip2 and LZMA2 in xz and 7z) that LZMA2 is simply, generally "the best" for compressing rather "noisy" data. Like binary programs and stuff. Naturally, I could look at the data and identify...

Fast random access isn't necessary. Being able to find, access and extract individual files (primarily on Microsoft Windows) is however, but if it takes e.g. 1-2 hours to find and extract an individual file, then that's okay. If it matters, the data set consists of 753047 files distributed over 90278 folders. The actual size of uncompressed data is 13381 GiB.

Hello, I'd like to compress a large data archive (HPE FTP mirror) with 7-Zip 22.01 for permanent storage. I would like to do this as efficiently as possible, meaning I'm targeting minimum archive size. However, since the data set is so large (over 13 TiB), I'd like to do it in a reasonable time frame, not multiple months as it would be on my local workstation with just 2-3 threads, being limited to 48 GiB of RAM. So I'm using a much larger machine. Hardware/Software: • Kernel Virtual Machine (RHV-H/RHV-M)...