7-Zip now reduces the number of working CPU threads for compression, if RAM size is not enough for compression with big LZMA2 dictionary.
7-Zip now can create and check "file.sha256" text files that contain the list of file names and SHA-256 checksums in format compatible with sha256sum program. 7-Zip can work with such checksum files as with archives, but these files don't contain real file data.
The context menu commands to create and test "sha256" files:
7z a -thash file.sha256 *.txt
7z t -thash file.sha256
7z t -thash -shd. file.sha256
New -shd{dir_path} switch to set the directory that is used to check files referenced by "file.sha256" file for "Test" operation. If -shd{dir_path} is not specified, 7-Zip uses the directory where "file.sha256" is stored.
New -xtd switch to exclude directory items from processing.
Note:
7-Zip's "Add to Archive" window now shows default options with asterisk character:
So there are some changes in code that selects default settings for each option. And 7-Zip tries to reduce the number of threads, if it can help to fit in RAM Size, if big dictionary size is specified.
Note also, that if you select default option with asterisk, that selected value will be not transferred to 7-zip compression engine. But 7-zip compression engine knows about default options, and it must use same values as shown to user in "Add to Archive" window.
Please write here, if you see any problem with all these new and changed features.
👍
5
Last edit: Igor Pavlov 2021-11-02
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Need an override of some sort to the memory check feature.
In Hyper-V Memory scales, so it should NOT ASSUME that there isn't enough memory
In NON-Hyper-V scenario's, it should NOT ASSUME that the max memory for settings chosen is ABSOLUTE, it ABSOLUTELY isn't!
A simple real world example I do on anything thats pre-compressed that I stick in my IT toolkit:
Download from AMD.com the latest Video driver (in my case today, 21.10.4)
Extract it to a folder
Recompress that folder it with self extracting feature and max compression settings
With 32GB of RAM Limited to 256MB word size according to the assumption being made with Vanilla 7Zip, unlimited (max word size) with Hex Modified 7Zip
EDIT: Also noted that at max word size with modified 7zip, 7zip did not use more then 13.5GB~ of Memory, definitely not the absurd number 7zip vanilla came up with as shown in the included picture
Picture results shown as well for validation.
Please add a checkbox in 7zip settings to ignore/bypass the memory check feature, and/or add a "try anyway" button to the memory check window.
You can set "Number of CPU threads" to default (asterisk) value.
Then change the dictionary size to big value.
If you need best compression ratio, you can set set "Number of CPU threads" to "2".
👍
1
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Igor, this does not solve the problem. Using only 2 threads to compress a terabyte will take forever. On a machine with lots of RAM and many CPU cores it is stupid to forcibly limit the usage just because 7-Zip cannot detect how much memory exists.
Machines with 12TB of RAM and 400+ CPU threads are common and cheap (~$50 USD / hour on google compute engine) and 7-Zip is completely unusable on these machines since the new memory limit checking is implemented because the API will not tell you how much physical memory exists - on modern hypervisors all memory is virtual.
Ideally, we should find a solution that does not require patches or forking of 7-Zip to continue using it. Can you please explain why you think that memory limit dialog is necessary? What problem does it solve? 7-Zip worked for many many years without this dialog and nobody suffered any major problems.
7-Zip now can create and check "file.sha256" text files that contain the list of file names and SHA-256 checksums in format compatible with sha256sum program.
That's great. Thank you.
But 7z h * -ba -scrcsha256 > txt.sha256 returns a file with "hashsum file_size file_name" structure.
Is there an option to not include "file_size" column there.
If there is not, could you implement such an option, please.
Last edit: ewild 2021-11-05
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I still suppose that 7z h is only additional method to get sha256 file. But the main (preffered command) is "add command":
7za-thasha.sha256*.txt7za-thasha.sha2562.txt
That way you can add and update new files to same sha256 file.
Note also that there are special cases, where file name in linux contains backslash, so you can get different lines in sha256 files with both methods. But only 7z a -thash must be compatible with sha256sum program in these special cases.
👍
1
Last edit: Igor Pavlov 2021-11-06
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Do you know what program did create that archive?
It shows that unix system was used. And 7-Zip thinks that it must use some unix charset (UTF-8) instead of windows charset.
👍
1
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
the encoding is lost after version 21.02
Version 21.01 everything is fine
That's the small problem as 21.01 source was not published and difference between 19.00 and 21.02 is rather big. Only the author can solve this as we don't have these sources.
Last edit: Sam Tansy 2021-11-20
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
If -xtd switch is used, the archive will contain all files even in subdirectories, but it will not contains separate directory items like these 2 items:
Date Time Attr Size Compressed Name
2021-11-01 20:14:12 D.... 0 0 1
2021-10-31 20:21:24 D.... 0 0 1\1
For example, such directory items has no hash sum . So we don't need such items in (sha256sum) hash files.
👍
2
Last edit: Igor Pavlov 2021-11-08
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
7-Zip 21.04 beta was released.
Download
7-Zip for 64-bit Windows x64:
https://7-zip.org/a/7z2104-x64.exe
7-Zip for 32-bit Windows x86:
https://7-zip.org/a/7z2104.exe
7-Zip for 64-bit Windows ARM64:
https://7-zip.org/a/7z2104-arm64.exe
7-Zip (console version) for 64-bit Linux x86-64 (AMD64):
https://7-zip.org/a/7z2104-linux-x64.tar.xz
7-Zip (console version) for 32-bit Linux x86:
https://7-zip.org/a/7z2104-linux-x86.tar.xz
7-Zip (console version) for 64-bit Linux ARM64:
https://7-zip.org/a/7z2104-linux-arm64.tar.xz
7-Zip (console version) for 64-bit macOS (x86-64 and ARM64):
https://7-zip.org/a/7z2104-mac.tar.xz
Source code:
https://7-zip.org/a/7z2104-src.7z
What's new after 7-Zip 21.03 beta:
sha256sumprogram. 7-Zip can work with such checksum files as with archives, but these files don't contain real file data.The context menu commands to create and test "sha256" files:
The commands for command line version:
7z a -thash file.sha256 *.txt 7z t -thash file.sha256 7z t -thash -shd. file.sha256-shd{dir_path}switch to set the directory that is used to check files referenced by "file.sha256" file for "Test" operation. If-shd{dir_path}is not specified, 7-Zip uses the directory where "file.sha256" is stored.-xtdswitch to exclude directory items from processing.Note:
7-Zip's "Add to Archive" window now shows default options with asterisk character:
So there are some changes in code that selects default settings for each option. And 7-Zip tries to reduce the number of threads, if it can help to fit in RAM Size, if big dictionary size is specified.
Note also, that if you select default option with asterisk, that selected value will be not transferred to 7-zip compression engine. But 7-zip compression engine knows about default options, and it must use same values as shown to user in "Add to Archive" window.
Please write here, if you see any problem with all these new and changed features.
Last edit: Igor Pavlov 2021-11-02
Thanks Igor.😃
Reporting again since it was reported in 21.03.
Need an override of some sort to the memory check feature.
In Hyper-V Memory scales, so it should NOT ASSUME that there isn't enough memory
In NON-Hyper-V scenario's, it should NOT ASSUME that the max memory for settings chosen is ABSOLUTE, it ABSOLUTELY isn't!
A simple real world example I do on anything thats pre-compressed that I stick in my IT toolkit:
Download from AMD.com the latest Video driver (in my case today, 21.10.4)
Extract it to a folder
Recompress that folder it with self extracting feature and max compression settings
With 32GB of RAM Limited to 256MB word size according to the assumption being made with Vanilla 7Zip, unlimited (max word size) with Hex Modified 7Zip
Results:
Original: 456 MB (478,168,392 bytes)
Vanilla 7Zip: 392 MB (411,276,239 bytes)
Modified 7Zip: 365 MB (383,065,860 bytes)
EDIT: Also noted that at max word size with modified 7zip, 7zip did not use more then 13.5GB~ of Memory, definitely not the absurd number 7zip vanilla came up with as shown in the included picture
Picture results shown as well for validation.
Please add a checkbox in 7zip settings to ignore/bypass the memory check feature, and/or add a "try anyway" button to the memory check window.
Last edit: Hammerfest 2021-11-04
You can set "Number of CPU threads" to default (asterisk) value.
Then change the dictionary size to big value.
If you need best compression ratio, you can set set "Number of CPU threads" to "2".
Igor, this does not solve the problem. Using only 2 threads to compress a terabyte will take forever. On a machine with lots of RAM and many CPU cores it is stupid to forcibly limit the usage just because 7-Zip cannot detect how much memory exists.
Machines with 12TB of RAM and 400+ CPU threads are common and cheap (~$50 USD / hour on google compute engine) and 7-Zip is completely unusable on these machines since the new memory limit checking is implemented because the API will not tell you how much physical memory exists - on modern hypervisors all memory is virtual.
Ideally, we should find a solution that does not require patches or forking of 7-Zip to continue using it. Can you please explain why you think that memory limit dialog is necessary? What problem does it solve? 7-Zip worked for many many years without this dialog and nobody suffered any major problems.
A small problem, 7-zip can't be uninstalled completely, whith leaves its name in the Windows programs list.
On Windows, the -thash option creates .sha256 files with an LF (unix) line break
Please correct this error at the end of the CR LF line
We use LF to get full byte-to-byte compatibility with
sha256sumprogram for these files. And notepad now supports LF files.There is another way to create CR LF file with undocumented switch
-slf:Also
-thashsupports-mflagsand-mm:t - method tag in each line
z - zero byte (NULL) instead of LF (as in sha256sum).
Last edit: Igor Pavlov 2021-11-05
That's great. Thank you.
But
7z h * -ba -scrcsha256 > txt.sha256returns a file with "hashsum file_size file_name" structure.Is there an option to not include "file_size" column there.
If there is not, could you implement such an option, please.
Last edit: ewild 2021-11-05
Read my comments about above about
-ba -slfhn.slf - "set list fields"
h - hash
s - size
n - name
-slf switch works perfectly doing exactly what has been requested.
Thank you very much!
Last edit: ewild 2021-11-06
I still suppose that
7z his only additional method to get sha256 file. But the main (preffered command) is "add command":That way you can add and update new files to same sha256 file.
Note also that there are special cases, where file name in linux contains backslash, so you can get different lines in sha256 files with both methods. But only
7z a -thashmust be compatible withsha256sumprogram in these special cases.Last edit: Igor Pavlov 2021-11-06
SSHFS drives are not listed in GUI.
http://www.secfs.net/winfsp/rel/
Hi, the encoding is lost after version 21.02
select some file in that folder inside zip archive and press Info button and show the windows here.
Also what do you have with internal Windows 10 Zip program?
inside
Last edit: Galed 2021-11-09
Do you know what program did create that archive?
It shows that unix system was used. And 7-Zip thinks that it must use some unix charset (UTF-8) instead of windows charset.
I can't say exactly how the archive was created. this archive is downloaded from the public procurement website
the problem remains at version 21.05
big request to fix it as it was at 21.01
Last edit: Galed 2021-11-23
That's the small problem as 21.01 source was not published and difference between 19.00 and 21.02 is rather big. Only the author can solve this as we don't have these sources.
Last edit: Sam Tansy 2021-11-20
Version 21.01 everything is fine
Last edit: Galed 2021-11-08
Igor,
-xtdswitch is used only when calculating the hash?Error trying to exclude a folder from processing
7z.exe a 1.7z -xtd"c:\2\folder" "c:\2\*"If
-xtdswitch is used, the archive will contain all files even in subdirectories, but it will not contains separate directory items like these 2 items:For example, such directory items has no hash sum . So we don't need such items in (sha256sum) hash files.
Last edit: Igor Pavlov 2021-11-08
Igor, thank you for reply.