7-Zip 21.03 beta

[str()]

Great to see this version is upgraded to beta status!

[teoberi]

GCC version: 11.2.0

cd CPP/7zip/Bundles/Alone2
make -f ../../cmpl_gcc_x64.mak

gcc -O2 -c -Wall -Werror -Wextra -Waddress -Waddress-of-packed-member -Waggressive-loop-optimizations -Wattributes -Wbool-compare -Wbool-operation -Wcast-align -Wcast-align=strict -Wcomment -Wdangling-else -Wdiv-by-zero -Wduplicated-branches -Wduplicated-cond -Wformat-contains-nul -Wimplicit-fallthrough=5 -Winit-self -Wint-in-bool-context -Wint-to-pointer-cast -Wunused -Wunused-macros -Wconversion -DNDEBUG -D_REENTRANT -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -fPIC -o b/g_x64/XzDec.o ../../../../C/XzDec.c
../../../../C/XzDec.c: In function ‘XzBlock_Parse’:
../../../../C/XzDec.c:776:3: error: this ‘if’ clause does not guard... [-Werror=misleading-indentation]
776 | if (s == 0) return SZ_ERROR_ARCHIVE; pos += s; }
| ^~
../../../../C/XzDec.c:823:5: note: in expansion of macro ‘READ_VARINT_AND_CHECK’
823 | READ_VARINT_AND_CHECK(header, pos, headerSize, &p->packSize);
| ^~~~~~~~~~~~~~~~~~~~~
../../../../C/XzDec.c:823:35: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the ‘if’
823 | READ_VARINT_AND_CHECK(header, pos, headerSize, &p->packSize);
| ^~~
../../../../C/XzDec.c:776:40: note: in definition of macro ‘READ_VARINT_AND_CHECK’
776 | if (s == 0) return SZ_ERROR_ARCHIVE; pos += s; }
| ^~~
../../../../C/XzDec.c:776:3: error: this ‘if’ clause does not guard... [-Werror=misleading-indentation]
776 | if (s == 0) return SZ_ERROR_ARCHIVE; pos += s; }
| ^~
../../../../C/XzDec.c:830:5: note: in expansion of macro ‘READ_VARINT_AND_CHECK’
830 | READ_VARINT_AND_CHECK(header, pos, headerSize, &p->unpackSize);
| ^~~~~~~~~~~~~~~~~~~~~
../../../../C/XzDec.c:830:35: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the ‘if’
830 | READ_VARINT_AND_CHECK(header, pos, headerSize, &p->unpackSize);
| ^~~
../../../../C/XzDec.c:776:40: note: in definition of macro ‘READ_VARINT_AND_CHECK’
776 | if (s == 0) return SZ_ERROR_ARCHIVE; pos += s; }
| ^~~
../../../../C/XzDec.c:776:3: error: this ‘if’ clause does not guard... [-Werror=misleading-indentation]
776 | if (s == 0) return SZ_ERROR_ARCHIVE; pos += s; }
| ^~
../../../../C/XzDec.c:837:5: note: in expansion of macro ‘READ_VARINT_AND_CHECK’
837 | READ_VARINT_AND_CHECK(header, pos, headerSize, &filter->id);
| ^~~~~~~~~~~~~~~~~~~~~
../../../../C/XzDec.c:837:35: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the ‘if’
837 | READ_VARINT_AND_CHECK(header, pos, headerSize, &filter->id);
| ^~~
../../../../C/XzDec.c:776:40: note: in definition of macro ‘READ_VARINT_AND_CHECK’
776 | if (s == 0) return SZ_ERROR_ARCHIVE; pos += s; }
| ^~~
../../../../C/XzDec.c:776:3: error: this ‘if’ clause does not guard... [-Werror=misleading-indentation]
776 | if (s == 0) return SZ_ERROR_ARCHIVE; pos += s; }
| ^~
../../../../C/XzDec.c:838:5: note: in expansion of macro ‘READ_VARINT_AND_CHECK’
838 | READ_VARINT_AND_CHECK(header, pos, headerSize, &size);
| ^~~~~~~~~~~~~~~~~~~~~
../../../../C/XzDec.c:838:35: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the ‘if’
838 | READ_VARINT_AND_CHECK(header, pos, headerSize, &size);
| ^~~
../../../../C/XzDec.c:776:40: note: in definition of macro ‘READ_VARINT_AND_CHECK’
776 | if (s == 0) return SZ_ERROR_ARCHIVE; pos += s; }
| ^~~
cc1: all warnings being treated as errors
make: *** [../../7zip_gcc.mak:1167: b/g_x64/XzDec.o] Error 1

[Paul]

Close the curly braces on the # define lines, i.e.

#define aaa \
{bbb \
ccc}

write like this:

#define aaa \
{bbb ccc}

in files:
7z2103-src/C/XzIn.c
7z2103-src/C/XzDec.c
7z2103-src/CPP/Common/MyCom.h
7z2103-src/CPP/7zip/Archive/XarHandler.cpp

[teoberi]

Thanks Paul!
Compile in this form as well:
Eg: 7z2103-src/C/XzDec.c

#define READ_VARINT_AND_CHECK(buf, pos, size, res) \
  { unsigned s = Xz_ReadVarInt(buf + pos, size - pos, res); if (s == 0) return SZ_ERROR_ARCHIVE; pos += s; }

[alex]

in the archive creation window, for 3 new dictionaries, the amount of memory for packaging should be shown not in GB, but in MB, like all old ones.

are there any plans (and deadlines) to add an option to setup an automatic, maximum possible dictionary size?

[Igor Pavlov]

2 digits with GB looks simpler than 5 digits MB.
"maximum possible dictionary size" is not good choice for usual user.

[alex]

1) to write a script, I need to know the exact consumption in mb with 2 streams, but I don't need beauty. if the user of the archiver needs beauty, he will setup something beautiful. winrar for example. 7zip is not chosen for its beauty.
2) with a dictionary of 1536mb and so 5 digits (16295), what's the difference? what will change if you make a display in mb for the new 3 sizes?
3) in the bench window, the list of dictionary sizes generally differs from the archive creation window for some reason. there is no 64kb value, and the new 3 sizes are indicated in GB (I did not compare in more detail).

[alex]

The maximum dictionary size for LZMA/LZMA2 compressing was increased to 4 GB (3840 MiB).

at the same time, in the window for adding files to the archive, the sizes of the dictionaries have the unit of measure "mb". should it be understood that a 3840mb dictionary in this window has the wrong unit of measurement, and it means "mib"?
or is the unit indicated correctly, but then its size is not 4gb but 3.75gb?

[Igor Pavlov]

MB means MiB (1024) in 7-Zip program.
I used "MB" fo many years. So it was not changed to more correct "MiB". I suppose Windows and Explorer also uses "MB" still.

"4 GB" in change log is just rounding number. By some reasons we can't use full "4 GiB" dictionary now for compression, and we need some reserved area that is 4096-3840=256 MiB now. If we reduce that reserved block, the compression will be slower. So it's just selected tradeoff between speed and maximum compression ratio. But 3840 MiB is almost as good as full 4 GiB dictionary in most case.

[alex]

I think it would be logical (and correct) to use the notation gib / mib / kib, since the measurement takes place in them...

[Umerov Artem]

in version 21.02 in ext4 filesystem image 7zip not view utf8 cyrillic codepage (can't detect). In 19.00 work fine

[Igor Pavlov]

Please attach some small example of such archive.

[Umerov Artem]

Ксожалению это не получится, файл весит 130 гигов. Повторить эту ошибку пока не удалось. Видимо есть имя файла, на котором происходит сбой в модуле определения кодировки. Советую тогда добавить выбор кодировки вручную у архивов

[Stuart]

Hi all

I'm using the 21.03 beta version of 7-Zip on 64-bit Windows 10, Version 21H1.
My computer has 8GB of RAM.

When compressing a small-sized folder (say total-size 300MB or-less) I used to be able to use the former maximum dictionary size of 1536MB, and it would complete without an issue.

Suddenly though, 7-Zip now won't allow me to go-above the 512MB dictionary size.
Even the next-one-up, 768MB, gives me this message:

Does anyone know when this change was made -- I cannot find any mention of it in any of the recent Release Notes. Also, is it possible within the Options to override this message? If not, would the developer consider adding such an override? If not, can anyone advise what the newest version of 7-Zip is before this change was made so I can revert back to that one?

Thanks

[Igor Pavlov]

7-Zip can create big load for swap, that can hang your system.
So 7-Zip now prevents from these dangerous compression modes.
If you have only 8GB, then use 512 MB dictionary.

[Stuart]

My pagefile is turned-off (to avoid excessive writes on my SSD), so there is no swap it can create a big load for ;)

Previously using a higher-dictionary was not an issue: 7-Zip would either tell me it cannot do it full-stop, or providing enough RAM was free it would begin and I would simply not open any-other apps while it was doing it's thing. It would often peak at around 7.7GB of RAM.

It's a shame therefore this "hard no" change has been introduced... so if I uninstall 21.03 and reinstall 21.02 can you confirm if that version is before this change was introduced, and if not what is the newest version I can rollback to please?

Thanks

[alex]

it would be nice to make an option cli to only use free physical memory.

[Obi-w00t]

Does the error take into account swap? If the user has 8GB RAM but 12GB swap will 7-Zip know it can use 20GB of memory?

[Igor Pavlov]

7-Zip looks physical RAM size.
Virtual memory cannot be fast for 7-Zip usage pattern.

[Obi-w00t]

It definitely won't be fast - but it would at least be functional. Perhaps as a compromise 7-Zip could check if the RAM and the RAM + Swap is enough for the operation.
+ If the RAM alone is enough, the operation passes through.
+ If the RAM + Swap is enough display a warning that the operation would be slow and suggest to decrease dictionary size, but allow the user continue if they choose.
+ If the RAM + Swap is not enough display the error, as it does now.

[post-factum]

It can be fast (see zram, which is commonly used on desktops nowadays). 7-zip should not make any assumptions about this.

[Halldor Fannar]

I'm seeing two issues with links in this release. Both issues happen when extracting an archive. It doesn't matter whether I recreate the archive (with the new -snl switch or use an archive made by an older 7z version).

1. Links are determined to be dangerous when they target files within the archive. I don't understand how that is dangerous. It seems the logic for determining whether the link is safe or not is flawed (it works sometimes but also fails often). Here is an example:
   ERROR: Dangerous symbolic link path was ignored : drive-t186ref-linux/targetfs/usr/share/zoneinfo/America/Havana : ../Cuba
   As you can see the relative link is just going up one level - should be determined as safe.

2. The extraction of the archive fails when it find links that are now determined to be unsafe. This breaks backwards compatibility with older versions of 7z (which don't fail extracting this archive). Here is an example: ERROR: Dangerous link path was ignored : drive-t186ref-linux/targetfs/usr/local/cuda : /usr/local/cuda-10.2. Now, I understand why this could be deemed as dangerous but I request that we add a new switch that changes these errors to a warning - allowing for backwards compatibility. That way these links are not created but the extraction is not deemed a failure. Another option is to allow the user to specify their appetite for "link danger level" completely. By default we would have the new behavior that 21.03 has - but allow the user via a switch to demote the error to a warning or a non-issue (and establish the link). The fact of the matter is that we have trusted packages within the company where I work and being able to establish these links, even when they go outside the package is quite useful. So allowing customers to have 100% backwards compatibility through a switch would be much appreciated. I'm basically arguing for a switch where I can specify an acceptance level 1: "external links must only issue a warning but not get created" or an acceptance level 2: "external links must not issue a warning or error and must be created". If this switch is not set we get the new behavior of "external links must be reported as errors and not be created, and fail the extraction process".

[Igor Pavlov]

1) maybe there are cases when we can create the sequence of "go to parent" links that will be dengerous.
the link "go to parent" breaks expectations for paths levels depth.
For example, when we extract path1\a.txt we must be sure that it's second-level path and is not first-level path.

2) there is -snld switch, that allows extracting of dangerous links. I'll add that switch to help file.

[Halldor Fannar]

I realize I put my reply on the wrong thread. Here it is in the right place :

Thanks for the prompt reply! I have tried the -snld switch and I'm observing the following:

I can now extract links that go up a directory level but remain inside the archive (thank you!)

Links that point outside the archive still generate an error message and cause the archive to not be extracted. This means that archives that I could extract with older versions of 7z are no longer usable with the new version :( It would be great if there was a switch that allowed these "outside" archive links to just issue a warning and not be created but the archive to be otherwise restored. Is this something you would be willing to entertain?

[Igor Pavlov]

Maybe we need additional mode to allow any links.
I'll think about it later.