7-Zip 21.03 beta

[post-factum]

Reported here.

I'd appreciate if you also consider talking to the uasm developer directly in case they reply in the GitHub issue.

Thanks.

[Obi-w00t]

UASM developers have said there is an option to disable stack alignment, based on this bug report: https://github.com/Terraspace/UASM/issues/118

I also checked the documentation here: http://www.terraspace.co.uk/uasm251_ext.pdf

I used to following to prefix the ASM file with the options we want:
sed -i -e '1i\OPTION FRAMEPRESERVEFLAGS:ON\nOPTION PROLOGUE:NONE\nOPTION EPILOGUE:NONE' Asm/x86/LzFindOpt.asm

When the Asm file is compiled it then produces a file exactly the same as the ASMC output.

Edit: in fact seeing as setting those options for all of the ASM files does not seem to produce any negative effects perhaps it is worthwhile testing if asmc can handle/ignore those options too?

[post-factum]

Yeah, thanks, I've updated my build with this, and it doesn't segfault any more.

I wonder if this is an uasm issue really.

[Obi-w00t]

It might depend on perspective - UASM implemented smart stack alignment and it is enabled by default. I think that is reasonable but in this case we don't want it, perhaps it trips up as we aren't creating an ASM program but rather using a snippet?

I couldn't find a way to disable those options on the command line so for now we have to prepend all the ASM files with those options. Either way ASMC has taken their Linux releases offline so it looks like we can't rely on ASMC and we have to ensure UASM behaves correctly for 7-Zip.

it may all be for nought - Debian has chosen not to use the ASM code at all to ensure portability and that may well be the easiest way to package for Linux.

[Paul]

Either way ASMC has taken their Linux releases offline so it looks like we can't rely on ASMC and we have to ensure UASM behaves correctly for 7-Zip.

But development for Linux continues: https://github.com/nidud/asmc/commits/master/source/asmc/linux and anyone can compile an asmc instance for themselves.

[teoberi]

Could you briefly explain how you can compile asmc in Linux?
For UASM it is interesting to watch:
https://github.com/Navegos/UASM/tree/dev-krad
If it is synchronized with UASM it could be a viable solution.

[Paul]

For my part, it will not be correct to post instructions for building ASMC for Linux, since I am not an ASMC developer, but I will post a text document in which I described the sequence of my actions to create ASMC for Linux, perhaps you will find a better solution.

[teoberi]

Indeed, ASMC Linux versions have started to appear again!
https://github.com/nidud/asmc/tree/master/source/asmc/linux/bin

Asmc Macro Assembler Version 2.32.55
Copyright (C) The Asmc Contributors. All Rights Reserved.
Portions Copyright (C) 1984-2002 Sybase, Inc. All Rights Reserved.

The only problem is how to build asmc for Linux, it is not very in line with the Linux path.
However, it can be used to compile the beta version for 7-Zip.

[Karlson2k]

Hi Igor.

Gentoo built-in QA check detected that all binaries with x86/amd64 assembler code has "stack executable" flags.
I've checked assembler-generated .o files.

$ readelf -S LzFindOpt.o
There are 7 section headers, starting at offset 0x40:

Section Headers:
  [Nr] Name              Type             Address           Offset           Size              EntSize          Flags  Link  Info  Align
  [ 0]                   NULL             0000000000000000  00000000        0000000000000000  0000000000000000           0     0     0
  [ 1] .text             PROGBITS         0000000000000000  00000200        0000000000000000  0000000000000000  AX       0     0     16
  [ 2] .data             PROGBITS         0000000000000000  00000200        0000000000000000  0000000000000000  WA       0     0     16
  [ 3] .text$LZFINDOPT   PROGBITS         0000000000000000  00000200        00000000000003a5  0000000000000000  AX       0     0     64
  [ 4] .shstrtab         STRTAB           0000000000000000  000005b0        0000000000000037  0000000000000000           0     0     1
  [ 5] .symtab           SYMTAB           0000000000000000  000005f0        0000000000000090  0000000000000018           6     5     4
  [ 6] .strtab           STRTAB           0000000000000000  00000680        0000000000000021  0000000000000000           0     0     1
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
  L (link order), O (extra OS processing required), G (group), T (TLS),
  C (compressed), x (unknown), o (OS specific), E (exclude),
  l (large), p (processor specific)

There is no ' .note.GNU-stack' section, so stack is marked as read/write/executable, which is security problem.

To fix it for arm64, you need to add to the end of *.S files the following lines:

#if defined(__linux__) && defined(__ELF__)
.section .note.GNU-stack,"",%progbits
#endif

I don't know any similar options for MASM files, so the only way to fix it for x86/amd64 is to add '-Wl,-z,noexecstack' option to link rule in makefile.

For more info see https://wiki.gentoo.org/wiki/Hardened/GNU_stack_quickstart

It is not 21.03-specific. The same is for 21.02.

[Karlson2k]

What's happen on Gentoo if the package is compiled without -noexestack:

 * QA Notice: The following files contain writable and executable sections
 *  Files with such sections will not work properly (or at all!) on some
 *  architectures/operating systems.  A bug should be filed at
 *  https://bugs.gentoo.org/ to make sure the issue is fixed.
 *  For more information, see:
 *
 *    https://wiki.gentoo.org/wiki/Hardened/GNU_stack_quickstart
 *
 *  Please include the following list of files in your report:
 *  Note: Bugs should be filed for the respective maintainers
 *  of the package in question and not hardened@gentoo.org.
 * RWX --- --- usr/bin/7zz

Actually use of -Wl,-z,noexecstack is a king of hack. The proper fix is to mark stack as "noexecutable" inside asm files. It can be done for 7-zip arm64 asm files, as they are using GAS syntax, but I not sure that any similar option exists for MASM files.

This is a security problem that can be easily fixed.

[teoberi]

cd CPP/7zip/Bundles/Alone2
make -f ../../cmpl_gcc_x64.mak

gcc -O2 -c -Wall -Werror -Wextra -Waddress -Waddress-of-packed-member -Waggressive-loop-optimizations -Wattributes -Wbool-compare -Wbool-operation -Wcast-align -Wcast-align=strict -Wcomment -Wdangling-else -Wdiv-by-zero -Wduplicated-branches -Wduplicated-cond -Wformat-contains-nul -Wimplicit-fallthrough=5 -Winit-self -Wint-in-bool-context -Wint-to-pointer-cast -Wunused -Wunused-macros -Wconversion -DNDEBUG -D_REENTRANT -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -fPIC -o b/g_x64/LzFind.o ../../../../C/LzFind.c
../../../../C/LzFind.c: In function ‘LzFindPrepare’:
../../../../C/LzFind.c:1604:13: note: ‘#pragma message: === LzFind SSE41’
1604 | #pragma message ("=== LzFind SSE41")
| ^~~~~~~
../../../../C/LzFind.c:1611:15: note: ‘#pragma message: === LzFind AVX2’
1611 | #pragma message ("=== LzFind AVX2")

7-Zip (z) 21.03 beta (x64) : Copyright (c) 1999-2021 Igor Pavlov : 2021-07-20
64-bit locale=en_US.UTF-8 Threads:4, ASM

[Igor Pavlov]

Is it problem?
I can disable these lines in source code.
What exact drawbacks of these pragma messages?

[Karlson2k]

These lines confuse package maintainers. It's unclear whether they indicate some error condition or just normal (I assume that they are just kind of debugging for compile-time).
I would suggest to either completely disable pragma messages without defined _DEBUG macro or make them more readable like Compiling SSE41 version of LzFind().

[teoberi]

My problem is that in 7-Zip (z) 21.02 alpha (x64) I had:

7-Zip (z) 21.02 alpha (x64) : Copyright (c) 1999-2021 Igor Pavlov : 2021-05-06
compiler: 10.3.0 GCC 10.3.0 64-bit locale=en_US.UTF-8 UTF8=+ use-UTF8=+ wchar_t=32-bit Files=64-bit Threads:4, Intel(R) Core(TM) i5-7500 CPU @ 3.40GHz (906E9),ASM,AES

In the 7-Zip (z) version 21.03 beta (x64) AES is missing.

[Igor Pavlov]

It's not missing.
7-Zip 21.03 just doesn't report that information about aes.
Most processors have aes. So there is no need to report about it in each run.
run

7z b "-mm=*" -mmt1 -md20 -mtic=30

and look speed numbers for aes lines.

[teoberi]

OK.

[AlexS]

what's for the switch -mtic=30 ?

[Igor Pavlov]

-mtic=30 sets a limit for about 2^30 cpu cycles for each benchmark workload. So you can get fast benchmarking with-md=20

There are also another undocumented switches for benchmark:
to test only 26 dictionary:

7z b -mdf=26 -mmt2

to benchmark for the data from real file file_name:

7z b -mfile=file_name

[AlexS]

Thanks!

[happyhappy88]

Can you add "Open Inside" "Open Inside*" "Open Inside#" to support commands l(List) e(Extract) x(eXtract) ?

[Igor Pavlov]

-t#
-t*

[happyhappy88]

Thank you.

[Pete]

FWIW, I converted the code for LzFindOpt.asm to NASM format. I incorporate it into my lrzip-next lzma-21.03beta branch. Attaching the nasm-converted source in case anyone finds it useful. It's x64 bit only.

[arhfk]

Dear Mr. Pavlov,

when do you decide to publish the current 7z version not only as an exe-file but also as an msi-file? version 21.03 beta has an msi-file (i found it somewhere on the page) but later versions did not have an msi-file like 21.01 alpha.

msi on windows 10 x64 all updates works a bit better than exe. i mean the installation is cleaner (no pseudo empty phantom lines in settings => programs but sadly the icon is missing)

[n00b]

you have all possible installers here:
https://www.7-zip.org/download.html
also msi for latest beta