I don't suppose somebody could tell me what the meaning of the following compression options: compression method, dictionary size, word size, solid block size and CPU threads.
And also, how I can use these options to get the best compression because the preset normal, maximum & ultra compression levels do not seem that different or to compress that well.
(Or in short could somebody please tell me the best compression settings and why they are best)
Thank you
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
If file size is your only criterion:
You want to select 7z, largest dictionary size you have RAM for, largest word size you have RAM for, and largest block size you can select.
The # of CPU threads doesn't change file size, but in most cases selecting "two" is likely to decrease time to complete.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Doesn't the compression depend on the type of file you are compressing? When I'm compressing music (mp3) the filesize stays exactly the same, and when compressing ISO images it only goes down by a few megabytes.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
MH,
You're correct.
Truly random data doesn't compress.
Data without redundancy also doesn't compress.
MP3 files are already compressed, and ISO files with compressed installers, for instance, also have little redundancy.
An ISO of a CD full of .TXT files would be expected to compress nicely.....
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Choosing a word size that is too large can increase the size of the archive but can easily be a negligible amount.
Test archive of mixed file types 19.5 MB (20,489,992 bytes)
<b>64 word size</b> , 64 MB dictionary, LZMA:
size 4.30 MB (4,513,706 bytes)
size on disk 4.30 MB (4,513,792 bytes)
<b> 273 word size</b> , 64 MB dictionary, LZMA:
size 4.30 MB (4,514,930 bytes)
size on disk 4.30 MB (4,517,888 bytes)
<i>>>> Have you seen another compression app outperform 7zip's performance when 7zip is set to Ultra?</i>
7zip is a great multipurpose compressor, but a specific purpose compressor used for the files it was designed to compress can make 7zip look like a complete joke. Soundfont compression software is a good example.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
"Choosing a word size that is too large can increase the size of the archive..."
This is not entirely true. If you scale the dictionary size along with the word size, you'll get a smaller archive. For example, here's some tests with the "Dictionary Size" setting maxed out (for me that's 512MB until I upgrade my RAM) and the "Word Size" Settings set accordingly:
Word Size: 8
Dictionary Size: 512MB
Total Size:1 511 001
Word Size: 12
Dictionary Size: 512MB
Total Size: 1 510 756
Word Size: 16
Dictionary Size: 512MB
Total Size: 1 510 581
Word Size: 64
Dictionary Size: 512MB
Total Size: 1 510 168
Word Size: 128
Dictionary Size: 512MB
Total Size: 1 510 053
Word Size: 192
Dictionary Size: 512MB
Total Size: 1 510 046
Word Size: 256
Dictionary Size: 512MB
Total Size: 1 510 057
NOTICE: The "Word Size" setting DECREASED the sized of the archive as long as I had a large enough "Dictionary Size" setting (as you can see, the file size went UP when I hit the 256 "Word Size" setting). And, I bet I could have maxed out the "Word Size" setting at "273" if I had enough RAM to set the Dictionary Size to "1024MB".
So the Moral of the Story is this: When setting the "Word Size" setting, make sure you scale it along with your "Dictionary Size" setting. You can only max out the "Word Size" setting if you are able to max out the "Dictionary Size" setting.
Last edit: Code Programmer 2015-05-19
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Sometimes choosing a smaller dictionary size (eg ultra @ 32 MB vs ultra @ 64 MB)
can also produce a negligibly smaller archive, with some savings in compression time
and large/huge savings in memory usage:
I'm already using max possible settings within the GUI controls that I can cram into my 18GB of RAM. But are there any other settings I can throw into "Parameters" field to increase compression even further regardless of the processing speed? Lets just say I want to get out maximum technically possible compression rate out of LZMA2. What should I use in the parameters fields for that?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hm, it doesn't work. I've tried lc4 and -lc4 within the parameter field. I'm using 7.38 BETA so that's probably the reason why it doesn't work. Anything for v9.38?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Oh crap, I'm sorry, it was my mistake. I had compression set to ZIP instead of 7z, that's why it didn't work.
I'll try it out and see how it performs. What exactly "lc4" does?
Btw Igor, why not adding one extra setting after "Ultra" named "Insane" or something like that. that would use absolutely max possible compression at expense of all the speed? I have CPU with 8 threads and tons of RAM, it's no real issue waiting few seconds longer for me to squeeze out best compression ratio, but would be a lot more convenient selecting such compression through existing drop down menu than remembering all the parameters (because you sometimes have to swithc to other compressions when making archives).
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I don't know the details, but, judging by its description, lc4 allocates 4 bits for some compression context. Default is 3 bits, maximum is 4 bits for LZMA2. If the context can become big enough for the data, allocating more bits will improve compression. Some data, however, benefits from lower lc values.
Since compression ratio has a very complex dependency on LZMA parameters, it may be difficult to create an "Insane" preset. What data do you usually compress - text, databases, images, multimedia? For text, lc8 pb0 (lc4 pb0 for LZMA2) usually gives best results.
Last edit: Shell 2015-05-29
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
We would like to do an extensive test for Winbuzzer comparing 7Zip with the other important compressing tools. How should we set word size and dictionary size settings when some other tools don´t offer the similar setting? I have the feeling the standard setting would underrate 7zips capacities and should be readjusted for today´s typical file sizes and filetypes.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
For best compression ratio:
format: 7z
level - ultra
method: lzma2
cpu threads = 2
Dictionary size - largest value allowed for RAM size at your computer.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I don't suppose somebody could tell me what the meaning of the following compression options: compression method, dictionary size, word size, solid block size and CPU threads.
And also, how I can use these options to get the best compression because the preset normal, maximum & ultra compression levels do not seem that different or to compress that well.
(Or in short could somebody please tell me the best compression settings and why they are best)
Thank you
Kevin,
If file size is your only criterion:
You want to select 7z, largest dictionary size you have RAM for, largest word size you have RAM for, and largest block size you can select.
The # of CPU threads doesn't change file size, but in most cases selecting "two" is likely to decrease time to complete.
Also, what do you mean "do not... compress that well" ?
Have you seen another compression app outperform 7zip's performance when 7zip is set to Ultra?
Your right. I just worded it wrong. I meant that sometimes I get (really) bad compression ratios.
Odd. For me, 7zip blows other compression apps out of the water like it was a shell from a 15 inch gun hitting a 13-foot-long fishing boat.
Doesn't the compression depend on the type of file you are compressing? When I'm compressing music (mp3) the filesize stays exactly the same, and when compressing ISO images it only goes down by a few megabytes.
MH,
You're correct.
Truly random data doesn't compress.
Data without redundancy also doesn't compress.
MP3 files are already compressed, and ISO files with compressed installers, for instance, also have little redundancy.
An ISO of a CD full of .TXT files would be expected to compress nicely.....
Ok thanks. That makes sense now as the majority of files I were compressing were actually MP3 files.
Thank you
Choosing a word size that is too large can increase the size of the archive but can easily be a negligible amount.
Test archive of mixed file types 19.5 MB (20,489,992 bytes)
<b>64 word size</b> , 64 MB dictionary, LZMA:
size 4.30 MB (4,513,706 bytes)
size on disk 4.30 MB (4,513,792 bytes)
<b> 273 word size</b> , 64 MB dictionary, LZMA:
size 4.30 MB (4,514,930 bytes)
size on disk 4.30 MB (4,517,888 bytes)
<i>>>> Have you seen another compression app outperform 7zip's performance when 7zip is set to Ultra?</i>
7zip is a great multipurpose compressor, but a specific purpose compressor used for the files it was designed to compress can make 7zip look like a complete joke. Soundfont compression software is a good example.
"Choosing a word size that is too large can increase the size of the archive..."
This is not entirely true. If you scale the dictionary size along with the word size, you'll get a smaller archive. For example, here's some tests with the "Dictionary Size" setting maxed out (for me that's 512MB until I upgrade my RAM) and the "Word Size" Settings set accordingly:
Word Size: 8
Dictionary Size: 512MB
Total Size:1 511 001
Word Size: 12
Dictionary Size: 512MB
Total Size: 1 510 756
Word Size: 16
Dictionary Size: 512MB
Total Size: 1 510 581
Word Size: 64
Dictionary Size: 512MB
Total Size: 1 510 168
Word Size: 128
Dictionary Size: 512MB
Total Size: 1 510 053
Word Size: 192
Dictionary Size: 512MB
Total Size: 1 510 046
Word Size: 256
Dictionary Size: 512MB
Total Size: 1 510 057
NOTICE: The "Word Size" setting DECREASED the sized of the archive as long as I had a large enough "Dictionary Size" setting (as you can see, the file size went UP when I hit the 256 "Word Size" setting). And, I bet I could have maxed out the "Word Size" setting at "273" if I had enough RAM to set the Dictionary Size to "1024MB".
So the Moral of the Story is this: When setting the "Word Size" setting, make sure you scale it along with your "Dictionary Size" setting. You can only max out the "Word Size" setting if you are able to max out the "Dictionary Size" setting.
Last edit: Code Programmer 2015-05-19
Sometimes choosing a smaller dictionary size (eg ultra @ 32 MB vs ultra @ 64 MB)
can also produce a negligibly smaller archive, with some savings in compression time
and large/huge savings in memory usage:
Source data: 53 292 905 bytes (firefox+profile)
7z 9.03, Ultra LZMA2 , word size 64 (default)
Dict size and archive size
16 15 639 442
24 15 624 209
32 15 622 269
48 15 624 209
64 15 622 290
Size on disk for all but Dict16MB is 15 626 240
Not the case with a slightly larger dataset (VLC 1.0.0, 75 034 543 bytes)
Ultra LZMA2, word 64
dict size archive size in MB compression time in secs
12 15,4 31
16 15,3 32
24 14,9 33
32 14,5 34
48 14,1 35
64 14,0 43 (actually size difference to Dict48: ~55KB)
Max LZMA2 , word 64
32 15,0 37
64 14,6 37
I'm already using max possible settings within the GUI controls that I can cram into my 18GB of RAM. But are there any other settings I can throw into "Parameters" field to increase compression even further regardless of the processing speed? Lets just say I want to get out maximum technically possible compression rate out of LZMA2. What should I use in the parameters fields for that?
7-Zip 15.03 alpha.
LZMA2 / 2 threads / 1536 MB dictionary.
You can try to write in parameters also:
lc4
and compare results with default (lc3).
Hm, it doesn't work. I've tried lc4 and -lc4 within the parameter field. I'm using 7.38 BETA so that's probably the reason why it doesn't work. Anything for v9.38?
lc4 must work. Check it again.
Oh crap, I'm sorry, it was my mistake. I had compression set to ZIP instead of 7z, that's why it didn't work.
I'll try it out and see how it performs. What exactly "lc4" does?
Btw Igor, why not adding one extra setting after "Ultra" named "Insane" or something like that. that would use absolutely max possible compression at expense of all the speed? I have CPU with 8 threads and tons of RAM, it's no real issue waiting few seconds longer for me to squeeze out best compression ratio, but would be a lot more convenient selecting such compression through existing drop down menu than remembering all the parameters (because you sometimes have to swithc to other compressions when making archives).
I don't know the details, but, judging by its description,
lc4allocates 4 bits for some compression context. Default is 3 bits, maximum is 4 bits for LZMA2. If the context can become big enough for the data, allocating more bits will improve compression. Some data, however, benefits from lowerlcvalues.Since compression ratio has a very complex dependency on LZMA parameters, it may be difficult to create an "Insane" preset. What data do you usually compress - text, databases, images, multimedia? For text,
lc8 pb0(lc4 pb0for LZMA2) usually gives best results.Last edit: Shell 2015-05-29
I've tried compressing game files and LC4 always gave me worse results.
As for the "Insane" profile, a general purpose max compression rate that ignores the speed entirely.
We would like to do an extensive test for Winbuzzer comparing 7Zip with the other important compressing tools. How should we set word size and dictionary size settings when some other tools don´t offer the similar setting? I have the feeling the standard setting would underrate 7zips capacities and should be readjusted for today´s typical file sizes and filetypes.
You can change "level" option.
For best compression ratio:
format: 7z
level - ultra
method: lzma2
cpu threads = 2
Dictionary size - largest value allowed for RAM size at your computer.
Just looked at 7z 16.04 help - cannot see 'level' - can see method -m but not 'ultra'.
7z a xx -mx filename works but no mention of ultra, or lzma2. What is line to implement your post?
Can the help as in 7z|more be improved to show the -m options more clearly? The only option mentioned is -mmt[N]
Chsnge to:
-m{Parameters} : set compression Method e.g. mentioning -mx ultra lzma2 mmt
TIA john
level - in GUI.
Commands for different compression ratio (64-bit version):
Thanks - so which of the is ultra, which lzma? Why not put this info in help under -m, it is only few extra characters or 1 line at most!
Otherwise how do the people who don't look in this forum know?
j
Also what is equivalent to this post for 32 bit 7z please? j.
ultra is -mx9.
lzma2 - is default method.
32-bit:
default dictionary size is 64MB for -mx9.
You can't compress with big dictionary (-md1024m) in 32-bit code.