Is there a way to use the 2 GB window size on zstd?
on zstd executable it is windowLog=31 or long=31
also it doesn't open files created this way to decode it needs --memory=2GB or something. gives "Data error"
2GB windows would be apreciated because it is so usefull for large files.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
we are planning on stabilizing the ZSTD_CCtx_setParameter() function in the next release, or the one after that, and then it will be safe to use.
Good to know it will be supported! Next version I would guess 3 or 6 months.
Would this allow to change other parameters too?
About the interface, is it a restriction for 7zip plugins the few choises avaiable?
I don't know the plugin UI limitations... is it possible to add the zstd compression to .7z or a ".tar.zst" archive format because to use this I would have to tar it then compress. This makes 7zip GUI less worth than on the fly tar it with pipes on the command line.
Here is a fast way I tested the windows size:
-make a 6650px square 126MB (little less than 2^27bytes) white BMP
-AES-256 it with 7z gets you a uncompressible 126MB .7z
-tar 2 copies of the file ->253MB .tar
-compress it with -1 to -21 gives 253MB , -22 gives 126MB.
-compress it with --long=27(still standard memory size) or larger gives 126MB.
I need sample for analysis.
Here I annexed a 12KB of a 128MB .BMP. Iit needs to be larger than 2^27 bytes or even if you set --long=28 it still uses smaller because there is no need.
from manpage:
Note: If windowLog is set to larger than 27, --long=windowLog or
--memory=windowSize needs to be passed to the decompressor.
Passing --memory=2GB always would be fine? It only affects decompression memory if it is needed.
The entire Tino's repository on GitHub is available again: https://github.com/mcmilk/7-Zip-zstd
Unfortunately it's not updated, but maybe somebody can fork and update it to 18.05? I know Dec's patches are good quality, I'm just concerned about missing UI for plugins.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Seems like libzstd.32.dll 1.4.4 does not support Windows XP SP3 (it raises an exception STATUS_ENTRYPOINT_NOT_FOUND in LoadLibraryEx()). Does it really use some APIs that are not present in XP?
Besides, 7-Zip officially supports Windows 2000, and Modern7z does not mention any system requirements. I think that this incompatibility, if it is not removed, should be mentioned in the plugin's description.
UPDATE: this DLL uses Synchronization API, namely condition variables, that appeared in Vista. I don't think it is easy to remove these calls. However, if these calls are implemented in a stub DLL, then system requirements can be lowered to XP SP2.
Last edit: Shell 2020-01-07
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I have found that ZSTD and FLZMA2 both use condition variables, so they require Windows Vista and newer OS. Other codecs do run on XP SP3 (and Brotli, as I guess, can even run on Windows 2000).
By the way, is it possible to check whether DLL can be loaded without side effects? MSDN says about LoadLibraryEx() that
If the function cannot find the module or one of its dependencies, the function fails.
In fact, however, NtRaiseHardError() is called, so the user is presented with a misleading popup (in the case of Modern7z, it says that 7-Zip has received Ctrl+C). Maybe it is worth loading external codecs with DONT_RESOLVE_DLL_REFERENCES and check PE header for OS/subsystem version before actually enabling the corresponding codec?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
This dll seems to be compiled using default MSVC settings, which are Vista compatible. To compile software for Windows XP it's not enough to install proper XP toolset (v120_xp for MSVC 2013, v140_xp or newer for MSVC 2015+) as it sets only minimal SUBSYSTEM version, but keeps default unchanged. As the linker uses default settings, the proper /SUBSYSTEM and /VERSION must be set manually.
See this discussion: https://stackoverflow.com/a/16088675
Some dll/exe files may work OK after changing /SUBSYSTEM and /VERSION with editbin tool provided in older MSVC versions. but it's not that case, I think.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
and similarly for FLZMA2. The very least that can be done is linking those functions dynamically (or with delayed load). This way, a codec may return an error on an unsupported system instead of throwing an exception.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I cannot produce ZST archive with Modern7z 1.8.5 (under 7z.exe 19.02 x86; under 7z.exe and 7zG.exe 19.00 x64). In console version, the -tzst switch produces
ERROR:
Unsupported archive type
GUI version also pops an error when I choose archive format ZSTD. Why?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
UPDATE: seems like that EH exception is thrown in 7z.exe, not in the DLLs. Perhaps, Modern7z.dll returns failure indicator. I can check the return values of exported functions if you tell me the necessary names.
Last edit: Shell 2020-01-10
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Have the same problem with command line. Checked under debugger - on my side everything is ok. Only Igor can say that is wrong in this case. But in FileManager everything works fine without any problem.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Yes, I finally managed to produce the archive in the GUI. The equality sign was necessary in the GUI (Parameters: x=22 instead of x22). But there is still one feature: both 7z.exe and 7zG.exe ban the value -mx=0 supported by ZSTD.
As for the console invocation, I'm looking forward to Igor Pavlov's response. Personally, I think that the problem sits in some 7-Zip API call: GUI and command line parser make Modern7z return different values for some reason.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
x=0 is banned by my code. Accordind to ZSTD docs: value 0 means default, which is controlled by ZSTD_CLEVEL_DEFAULT. But in context of 7zip x means compression level. Low value - low level, high value - high level. X parameter has no default value.
👍
1
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Does Modern7z support the parameters ZSTD_strategy, ZSTD_c_windowLog (dictionary size) or ZSTD_c_targetLength (fast bytes) now? The previous post about window size is dated 1.5 years ago. I've seen claims that parameter d could be set for ZSTD, but I don't know whether this applies to your codec.
Last edit: Shell 2020-04-21
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
ZSTD_strategy - "a" parameter, ZSTD_c_windowLog - "d" parameter, ZSTD_c_targetLength currently is not supported, but it is easy to add it. I can add support of all ZSTD paramerters, but 7-Zip plugin api has a limited number of parameters that can be passed to the codec.
👍
1
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Thank you for the answer. I failed to use a and d first time, but now I spotted that the folder "Codecs" was not on the path. Please mention those parameters in documentation when you're going to revise it.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
For example WinCryptHashers.xx.dll + WinCryptHashers.ini. ReadMe.txt : "WinCryptHashers.ini is a plain-text file that allows you to configure the
plugin."
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Modern7z allows you to extract LZ4 files.
Is there a way to use the 2 GB window size on zstd?
on zstd executable it is windowLog=31 or long=31
also it doesn't open files created this way to decode it needs --memory=2GB or something. gives "Data error"
2GB windows would be apreciated because it is so usefull for large files.
This function is dll has experimental status. Comment from github:
I will add this possibility when ZSTD_CCtx_setParameter is stable.
I need sample for analysis.
Good to know it will be supported! Next version I would guess 3 or 6 months.
Would this allow to change other parameters too?
About the interface, is it a restriction for 7zip plugins the few choises avaiable?
I don't know the plugin UI limitations... is it possible to add the zstd compression to .7z or a ".tar.zst" archive format because to use this I would have to tar it then compress. This makes 7zip GUI less worth than on the fly tar it with pipes on the command line.
Here is a fast way I tested the windows size:
-make a 6650px square 126MB (little less than 2^27bytes) white BMP
-AES-256 it with 7z gets you a uncompressible 126MB .7z
-tar 2 copies of the file ->253MB .tar
-compress it with -1 to -21 gives 253MB , -22 gives 126MB.
-compress it with --long=27(still standard memory size) or larger gives 126MB.
Here I annexed a 12KB of a 128MB .BMP. Iit needs to be larger than 2^27 bytes or even if you set --long=28 it still uses smaller because there is no need.
from manpage:
Passing --memory=2GB always would be fine? It only affects decompression memory if it is needed.
Last edit: Fred 2018-07-10
Yes
Plugins have no access to UI at all. Maybe Igor will improve compression dialog in someday.
The entire Tino's repository on GitHub is available again:
https://github.com/mcmilk/7-Zip-zstd
Unfortunately it's not updated, but maybe somebody can fork and update it to 18.05? I know Dec's patches are good quality, I'm just concerned about missing UI for plugins.
Hello,
I just updated 7-Zip-zstd to version 18.05 with Zstandard 1.3.7 ;-)
Hi Tino! Good to hear from you. Hope you're doing well. Thanks for the update!
Seems like libzstd.32.dll 1.4.4 does not support Windows XP SP3 (it raises an exception STATUS_ENTRYPOINT_NOT_FOUND in
LoadLibraryEx()). Does it really use some APIs that are not present in XP?Besides, 7-Zip officially supports Windows 2000, and Modern7z does not mention any system requirements. I think that this incompatibility, if it is not removed, should be mentioned in the plugin's description.
UPDATE: this DLL uses Synchronization API, namely condition variables, that appeared in Vista. I don't think it is easy to remove these calls. However, if these calls are implemented in a stub DLL, then system requirements can be lowered to XP SP2.
Last edit: Shell 2020-01-07
It is better to ask ZSTD developers about it.
Ok, I will add notice.
I have found that ZSTD and FLZMA2 both use condition variables, so they require Windows Vista and newer OS. Other codecs do run on XP SP3 (and Brotli, as I guess, can even run on Windows 2000).
By the way, is it possible to check whether DLL can be loaded without side effects? MSDN says about
LoadLibraryEx()thatIn fact, however,
NtRaiseHardError()is called, so the user is presented with a misleading popup (in the case of Modern7z, it says that 7-Zip has received Ctrl+C). Maybe it is worth loading external codecs withDONT_RESOLVE_DLL_REFERENCESand check PE header for OS/subsystem version before actually enabling the corresponding codec?This dll seems to be compiled using default MSVC settings, which are Vista compatible. To compile software for Windows XP it's not enough to install proper XP toolset (v120_xp for MSVC 2013, v140_xp or newer for MSVC 2015+) as it sets only minimal SUBSYSTEM version, but keeps default unchanged. As the linker uses default settings, the proper /SUBSYSTEM and /VERSION must be set manually.
See this discussion: https://stackoverflow.com/a/16088675
Some dll/exe files may work OK after changing /SUBSYSTEM and /VERSION with editbin tool provided in older MSVC versions. but it's not that case, I think.
It is not the case because those codecs use new API functions:
and similarly for FLZMA2. The very least that can be done is linking those functions dynamically (or with delayed load). This way, a codec may return an error on an unsupported system instead of throwing an exception.
I cannot produce ZST archive with Modern7z 1.8.5 (under 7z.exe 19.02 x86; under 7z.exe and 7zG.exe 19.00 x64). In console version, the
-tzstswitch producesGUI version also pops an error when I choose archive format ZSTD. Why?
Cannot reproduce. Need more details. OS version? Mayby some screenshots?
Running through NTSD under Windows Server 2016 (x64):
and the process terminates.
UPDATE: seems like that EH exception is thrown in 7z.exe, not in the DLLs. Perhaps, Modern7z.dll returns failure indicator. I can check the return values of exported functions if you tell me the necessary names.
Last edit: Shell 2020-01-10
Have the same problem with command line. Checked under debugger - on my side everything is ok. Only Igor can say that is wrong in this case. But in FileManager everything works fine without any problem.
Yes, I finally managed to produce the archive in the GUI. The equality sign was necessary in the GUI (Parameters: x=22 instead of x22). But there is still one feature: both 7z.exe and 7zG.exe ban the value
-mx=0supported by ZSTD.As for the console invocation, I'm looking forward to Igor Pavlov's response. Personally, I think that the problem sits in some 7-Zip API call: GUI and command line parser make Modern7z return different values for some reason.
x=0 is banned by my code. Accordind to ZSTD docs: value 0 means default, which is controlled by ZSTD_CLEVEL_DEFAULT. But in context of 7zip x means compression level. Low value - low level, high value - high level. X parameter has no default value.
Does Modern7z support the parameters ZSTD_strategy, ZSTD_c_windowLog (dictionary size) or ZSTD_c_targetLength (fast bytes) now? The previous post about window size is dated 1.5 years ago. I've seen claims that parameter
dcould be set for ZSTD, but I don't know whether this applies to your codec.Last edit: Shell 2020-04-21
ZSTD_strategy - "a" parameter, ZSTD_c_windowLog - "d" parameter, ZSTD_c_targetLength currently is not supported, but it is easy to add it. I can add support of all ZSTD paramerters, but 7-Zip plugin api has a limited number of parameters that can be passed to the codec.
Thank you for the answer. I failed to use
aanddfirst time, but now I spotted that the folder "Codecs" was not on the path. Please mention those parameters in documentation when you're going to revise it.Use .ini file. If I think correctly, there is no limit to that. For example x=30-100 user definied lines. (23-29 reserved for original codec.)
Sorry but I don't understand your idea.
For example WinCryptHashers.xx.dll + WinCryptHashers.ini. ReadMe.txt : "WinCryptHashers.ini is a plain-text file that allows you to configure the
plugin."