p7zip / Feature Requests / #43 7za 16.02 memory failure (where 9.20 works)

Igor Pavlov - 2016-08-03

7-zip 9.20 used lzma with 2 threads.
7-zip 16 uses lzma2 with more threads (8 in your case).
Each 2 threads require 700-800 MB in -mx9 mode.

You can reduce the number of threads:

-mmt2 -mx9

Also you can use 64-bit version of 7-zip, if your system is 64-bit.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Charlie Root - 2016-08-03

This worked at -mmt5 (failed at 6). I will probably use 4 for my critical backups.

$ time 7za a -mmt5 -mx=9 -bd test2.7z test.dat

7-Zip (a) [32] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,32 bits,8 CPUs Intel(R) Xeon(R) CPU E5-2609 0 @ 2.40GHz (206D7),ASM,AES-NI)

Scanning the drive:
1 file, 2146443264 bytes (2048 MiB)

Creating archive: test2.7z

Items to compress: 1

Files read from disk: 1
Archive size: 251642805 bytes (240 MiB)
Everything is Ok

real 9m20.381s
user 33m58.043s
sys 0m6.722s

$ ll test2.7z
-rw-r--r-- 1 oracle dba 251642805 Aug 3 11:02 test2.7z ###not quite as good as 9.20

This raises a few interesting points.

Your binary distribution offered here for download is 32-bit only - 64-bit binaries are not provided (this is all right for me in this case):

$ tar xvjf p7zip_16.02_x86_linux_bin.tar.bz2 p7zip_16.02/bin
p7zip_16.02/bin/
p7zip_16.02/bin/7za
p7zip_16.02/bin/7zCon.sfx
p7zip_16.02/bin/Codecs/
p7zip_16.02/bin/Codecs/Rar.so
p7zip_16.02/bin/7z.so
p7zip_16.02/bin/7z
p7zip_16.02/bin/7zr

$ file p7zip_16.02/bin/7za
p7zip_16.02/bin/7za: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.0.30, dynamically linked (uses shared libs), stripped

$ ldd p7zip_16.02/bin/7za
linux-gate.so.1 => (0x00856000)
libpthread.so.0 => /lib/libpthread.so.0 (0x0049d000)
libc.so.6 => /lib/libc.so.6 (0x002f7000)
/lib/ld-linux.so.2 (0x002d8000)

$ file busybox-i686
busybox-i686: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), statically linked, stripped

What you've mentioned above is key information for using your 32-bit binary distribution. It would be nice to have a "MAXIMUM_COMPRESSION.txt" file in the distribution that details best settings for size and size/speed that includes details for both 32/64-bit. If we had that, then extra program features to automatically reduce the threadcount would not be necessary from an "rtfm" perspective.

The 32-bit binary that you provide is dynamically linked against glibc. It would be nice if the binary was statically linked against a smaller libc(++), similar to busybox (https://busybox.net/downloads/binaries/ - shown above) and Aboriginal dropbear (http://landley.net/aboriginal/downloads/old/binaries/1.4.3/extras/). This would make it able to run on more systems, assuming the new libc works with your threading model. After a few searches, it appears that uClibc++ is your best bet: https://cxx.uclibc.org/

An ARM binary prepared this way will run cleanly on Android, and a lot of other ARM systems. I would think that you would prefer uClibc(++) over Bionic.

I could try to generate x86/x64/arm binaries for you if there is interest.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Igor Pavlov - 2016-08-04

I'm developer of 7-Zip (windows version).
p7zip is maintained by another developer.
I don't know why there is no 64-bit download at SF.

For best compression ratio for your file you can try:

-mx -mmt2

or

-mx -mmt2 -md96m

but

-mx -mmt4

is 2 times faster.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Charlie Root - 2016-08-04

With the 32-bit distribution binary using -md96m, -mmt3 will run, but -mmt4 will not. This now compresses better than 9.20. FYI, Linux allows a 3GB 32-bit userland.

I'll download the ARM uClibc root image and see if it will build 7za this weekend. That might be fun.

$ 7za a -mmt4 -md96m -mx=9 -bd test2.7z test.dat

7-Zip (a) [32] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,32 bits,8 CPUs Intel(R) Xeon(R) CPU E5-2609 0 @ 2.40GHz (206D7),ASM,AES-NI)

Scanning the drive:
1 file, 2146443264 bytes (2048 MiB)

Creating archive: test2.7z

Items to compress: 1

ERROR: Can't allocate required memory!

$ time 7za a -mmt3 -md96m -mx=9 -bd test2.7z test.dat

7-Zip (a) [32] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,32 bits,8 CPUs Intel(R) Xeon(R) CPU E5-2609 0 @ 2.40GHz (206D7),ASM,AES-NI)

Scanning the drive:
1 file, 2146443264 bytes (2048 MiB)

Creating archive: test2.7z

Items to compress: 1

Files read from disk: 1
Archive size: 248550709 bytes (238 MiB)
Everything is Ok

real 18m33.653s
user 36m0.919s
sys 0m6.515s

$ ll test2.7z
-rw-r--r-- 1 oracle dba 248550709 Aug 4 09:28 test2.7z

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Charlie Root - 2016-08-04

I was able to make my own 64-bit binary from the source distribution. That process seems simple enough.

$ cp -v makefile.linux_amd64_asm makefile.linux
$ make all_test
...
$ file 7za.16.02
7za.16.02: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=325a4739d0a0ddfe41bcdb72d45e051f53dbf32f, stripped

$ ldd 7za.16.02
linux-vdso.so.1 => (0x00007ffd2dd7d000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f3886104000)
libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f3885dfb000)
libm.so.6 => /lib64/libm.so.6 (0x00007f3885af9000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f38858e3000)
libc.so.6 => /lib64/libc.so.6 (0x00007f3885524000)
/lib64/ld-linux-x86-64.so.2 (0x00007f3886320000)

$ ./7za.16.02

7-Zip (a) [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=en_US.utf8,Utf16=on,HugeFiles=on,64 bits,8 CPUs x64)

Usage: 7za [<switches>...] <archive_name> [<file_names>...]
...

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Igor Pavlov - 2016-08-05

With 64-bit version and 8 GB of RAM, you can use big dictionary:

-mx -mmt2 -md512m -mx -mmt4 -md256m -mx -mmt8 -md128m
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Charlie Root - 2016-08-05

The Linux 32-bit binary is "advertised" as statically-linked:

$ tar xvjf p7zip_16.02_x86_linux_bin.tar.bz2 p7zip_16.02/README
p7zip_16.02/README

$ awk '/"bin"/,/statically/' p7zip_16.02/README
If you have downloaded the "bin" package,
use directly the program bin/7za. (tested on Debian 3, Ubuntu 14.10)
As the program is statically linked, its should run on many x86 or amd64 linux.

If this were so, then I would be able to run it in the chroot below.

$ mkdir -p chroot/bin
$ mv 7za.16.02 busybox-i686 chroot/bin
$ cd chroot/bin/
$ ./busybox-i686 --list | awk '{print "ln -s busybox-i686 " $0}' | sh
$ export SHELL=/bin/sh
$ chroot /root/chroot
/ # /bin/7za.16.02
/bin/sh: /bin/7za.16.02: not found
/ # ls -l /bin/7za.16.02
-rwxr-xr-x 1 0 0 1639708 Aug 5 20:32 /bin/7za.16.02

I'll see if I can make a true static binary over the weekend. This will depend upon C++ support in the uClibc development images.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Charlie Root - 2016-08-09

It appears that http://buildroot.org has the best development environment for static binaries of all architecure types. I tried to set this up over the weekend, but it did not include gcc in my filesystem for some reason. I am assuming that the Linux distribution binary came from the Intel icc compiler, which buildroot doesn't use.

On my 32GB x64 system, this is the best/fastest call so far: 7za a -mx -mmt8 -md512m
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

7za 16.02 memory failure (where 9.20 works)

Group

Searches

Help

#43 7za 16.02 memory failure (where 9.20 works)

Discussion