Windows 10. Version 1709 (Build 16299.309)
CPU i5-7600K
(1) 7zG b -mmt1 -slp -md21
allways crash
(2) 7z b -mmt1 -slp -md19 -bt 10 >19.txt 2>19_2.txt
can work some time after reboot, but after (1) and (3) - crash
(3) 7z b -mmt1 -slp -md22 -bt 10 >22.txt 2>22_2.txt
allways crash
BTW running the bench with/without the slp option I cannot get very reliable numbers to say that large pages give an improvement in speed.
Benchmark with dict 24: and 25: give advantages in compression (about 10%), other have same performances and better decompression performance without slp.
Last edit: Pach Ugo 2018-04-06
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
1) 7-zip creates a security issue when enabling the 'Large memory pages' option since it's adding the SeLockMemoryPrivilege to the current user account using LsaAddAccountRights thus allowing every application (even sandboxed processes) access to that privilege... Stop assigning privileges to the account and instead adjust the privileges of the current process token or ShellExecute with RunAs to execute as administrator and obtain the privilege.
There are some other issues but the main one is "applications should avoid making repeated large-page allocations and instead allocate all large pages one time, at startup."
The "7zG b -mmt1 -slp -md21" command allocated and freed a total of 190 large allocations over approx 18 minutes.
"7z b -mmt1 -slp -md19 -bt 10" allocated and freed a total of 30 large allocations during runtime over approx four minutes.
"7z b -mmt1 -slp -md22 -bt 10" allocated and freed a total of 20 large allocations over approx two minutes.
I'm willing to bet that if 7zip created just one allocation at startup and used that thoughout execution (instead of allocating and freeing the memory 240 times) that it would solve this problem.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
"Large pages" is good feature.
Some programs can work 10-15% faster with large pages.
And Linux system uses Transparent Huge Pages (THP) for any application. So windows programs can work faster under wine in Linux than same programs under Windows without large pages.
If we want to get best available performance, we still need good way to use large pages in Windows too.
In the ideal case we need "Transparent Large Pages" in Windows or some simplified ways to allocate large pages without additional Privileges. But Windows doesn't implement it.
Instead of that, Windows implements complicated way for large pages. That way is so complicated, so most programs for Windows ignore that feature.
And it could be the reason of that BUG in Windows 10. If almost nobody uses some feature of Windows, than nobody reports about BUG in that feature. And probably that bug was introduced in 2017 and was not fixed at least for about one year.
Now I still need some information:
- confirmation from microsoft that there was BUG in Windows 10. ).
- full list of affected systems and revisions.
- are Windows server versions affected too?
So I will be able to add checks to program and disable "Large Pages" feature for these affected systems.
Last edit: Igor Pavlov 2018-04-10
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
No errors.
Win 10 Pro 10.0.16299.371
Intel Core i7-3520M CPU
Ran all three commands seperately without fail. Ran last two commands again simultaneously (just because I like to try to break the system) without fail.
Have you noticed specific success of Intel(R) Core(TM) i7-3xxx & i5-3xxx processors (CPUID of both is (306A9) )?
So the main reason might be in hardware , not OS or other software?
And one more strange thing - the best performance could be achived at 8Mb size dictionary only - not more, not less - is it connected in any way with algorithm implemented?
Last edit: Mikhail Rokhin 2018-04-16
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
7zG b -mmt1 -slp -md1 -bt 10 - OK
...
7zG b -mmt1 -slp -md18 -bt 10 -OK
7zG b -mmt1 -slp -md19 -bt 10 - OK
Failed, when started with the tests first, but after numerous other tests these one are completed successfully - strange behaviour... very strange... smth with rights assign?
---Spontaneous failures ??? ---
7zG b -mmt1 -slp -md20 -bt 10 - 7zG.exe Decoding error / OK after numerous other -mdXX tests
7z b -mmt1 -slp -md20 -bt 10 - OK - Why does GUI Fail ?
7zG b -mmt1 -slp -md21 - 7zG.exe Decoding error / Fails & Closed / OK after numerous other -mdXX tests
7z b -mmt1 -slp -md21 -bt 10 - OK - Why does GUI Fail ?
7z b -mmt1 -slp -md19 -bt 10 >19.txt 2>19_2.txt - OK
7zG b -mmt1 -slp -md19 -bt 10 - OK
7zG b -mmt1 -slp -md23 -bt 10 - OK
7z b -mmt1 -slp -md22 -bt 10 >22.txt 2>22_2.txt - OK
7z b -mmt1 -slp -md22 -bt 10 - OK
7zG b -mmt1 -slp -md23 -bt 10 - OK
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Microsoft Windows 10 x64 Home via VirtualBox 5.2.8
1803 [Version 10.0.17134.1]
7zG b -mmt1 -slp -md20 -bt 10 - OK for Adm; OK for User
7z b -mmt1 -slp -md20 -bt 10 - OK for Adm; OK for User
7zG b -mmt1 -slp -md21 - OK for Adm; OK for User
7z b -mmt1 -slp -md21 -bt 10 - OK for Adm; OK for User
7z b -mmt1 -slp -md19 -bt 10 >19.txt 2>19_2.txt - OK for Adm; OK for User
7zG b -mmt1 -slp -md19 -bt 10 - OK for Adm; OK for User
7z b -mmt1 -slp -md22 -bt 10 >22.txt 2>22_2.txt - OK for Adm; OK for User
7zG b -mmt1 -slp -md22 -bt 10 - OK for Adm; OK for User
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
ASLR, SEHOP, CF-Guard etc. are also enabled on my system, but working fine nevertheless.
By the way... 1803 is RS4 aka RTM (Build 17133.1). Have you tested it with Build 10.0.17618.1000? This is a preview build of RS5 (SKIP).
Last edit: Uwe Ruch 2018-04-02
Last edit: 0815 2018-04-25
Note also that if we redirect output, it increases the chance to get error in windows 1709:
First command can work without error.
But second command can crash.
Last edit: 0815 2018-04-25
Windows 10.0.14939.1944 (1607)
i7 3770, 16 GiB RAM
Large Pages are only used when run as Admin
No exception or other errors occured while running the benchmarks.
Windows 10. Version 1709 (Build 16299.309)
CPU i5-7600K
(1) 7zG b -mmt1 -slp -md21
allways crash
(2) 7z b -mmt1 -slp -md19 -bt 10 >19.txt 2>19_2.txt
can work some time after reboot, but after (1) and (3) - crash
(3) 7z b -mmt1 -slp -md22 -bt 10 >22.txt 2>22_2.txt
allways crash
Last edit: Vladimir Presnyakov 2018-04-05
Windows 10 x64 1709 16299.334
Intel i7 6500U
7-Zip 18.01 x64
7zG b -mmt1 -slp -md21 crashes the app
7z b -mmt1 -slp -md19 -bt 10 >19.txt 2>19_2.txt: completes ok
7z b -mmt1 -slp -md22 -bt 10 >22.txt 2>22_2.txt: crash after 10/15 secs, no output in the redirected files
No hangs when the test starts, all applications run smoothly
BTW running the bench with/without the slp option I cannot get very reliable numbers to say that large pages give an improvement in speed.
Benchmark with dict 24: and 25: give advantages in compression (about 10%), other have same performances and better decompression performance without slp.
Last edit: Pach Ugo 2018-04-06
win 8.1 ok.
need admin rights.
Last edit: alex 2018-04-09
@Igor Pavlov
1) 7-zip creates a security issue when enabling the 'Large memory pages' option since it's adding the SeLockMemoryPrivilege to the current user account using LsaAddAccountRights thus allowing every application (even sandboxed processes) access to that privilege... Stop assigning privileges to the account and instead adjust the privileges of the current process token or ShellExecute with RunAs to execute as administrator and obtain the privilege.
2) MEM_LARGE_PAGES is designed for server applications (winrar and winzip don't use this option and neither should 7zip) and the documentation is clear about a few issues: https://msdn.microsoft.com/en-us/library/windows/desktop/aa366720.aspx
There are some other issues but the main one is "applications should avoid making repeated large-page allocations and instead allocate all large pages one time, at startup."
The "7zG b -mmt1 -slp -md21" command allocated and freed a total of 190 large allocations over approx 18 minutes.
"7z b -mmt1 -slp -md19 -bt 10" allocated and freed a total of 30 large allocations during runtime over approx four minutes.
"7z b -mmt1 -slp -md22 -bt 10" allocated and freed a total of 20 large allocations over approx two minutes.
I'm willing to bet that if 7zip created just one allocation at startup and used that thoughout execution (instead of allocating and freeing the memory 240 times) that it would solve this problem.
"Large pages" is good feature.
Some programs can work 10-15% faster with large pages.
And Linux system uses Transparent Huge Pages (THP) for any application. So windows programs can work faster under wine in Linux than same programs under Windows without large pages.
If we want to get best available performance, we still need good way to use large pages in Windows too.
In the ideal case we need "Transparent Large Pages" in Windows or some simplified ways to allocate large pages without additional Privileges. But Windows doesn't implement it.
Instead of that, Windows implements complicated way for large pages. That way is so complicated, so most programs for Windows ignore that feature.
And it could be the reason of that BUG in Windows 10. If almost nobody uses some feature of Windows, than nobody reports about BUG in that feature. And probably that bug was introduced in 2017 and was not fixed at least for about one year.
Now I still need some information:
- confirmation from microsoft that there was BUG in Windows 10. ).
- full list of affected systems and revisions.
- are Windows server versions affected too?
So I will be able to add checks to program and disable "Large Pages" feature for these affected systems.
Last edit: Igor Pavlov 2018-04-10
Last edit: 0815 2018-04-25
P.S, Did you know that Windows 10, use memory compression,
Last edit: 0815 2018-04-25
Windows x64 10.0.17123.1, 7-Zip 18.03 x64
No errors, stderr files are empty
Win 10 x64 1709 with April 2018 Cumulative, build 16299.371
7-Zip 18.01 x64
CPU: i7-2760QM
RAM: 16GB
Turn on LP (admin mode 7zfm)
Reboot
Open user mode powershell
.\7zG b -mmt1 -slp -md21
"Decoding Error"
.\7z b -mmt1 -slp -md19 -bt 10 >d:\19.txt 2>d:\19_2.txt
Finished successfully, 19.txt shows "LP" in RAM size / usage
.\7z b -mmt1 -slp -md22 -bt 10 >d:\22.txt 2>d:\22_2.txt
"7-Zip Console has stopped working"
Reboot
Turn off LP (admin mode 7zfm)
Reboot
No errors.
Win 10 Pro 10.0.16299.371
Intel Core i7-3520M CPU
Ran all three commands seperately without fail. Ran last two commands again simultaneously (just because I like to try to break the system) without fail.
About 16299.371
There are 3 "ERROR" reports and only one "OK" report from you.
Please reboot your system and try tests again.
Also you can check that large pages really worked.
You must see some difference in "Compressing Rating MIPS" for the following commands:
Windows 10 Pro 64Bit, Version 1709, OS Build: 16299.371
7Zip 18.01 64Bit.
System crashed with blue screen error "write to not writable memory" or something similar.
That happened at first command 7zG.exe
Windows 10 pro 64Bit, Version 1709, OS Build: 16299.371
7Zip 18.01 64Bit
Intel i7-4700MQ
Test 1 and 3 ran without an error, 19_2.txt stated "Decoding error"
Last edit: Gene Theller 2018-04-16
Windows 10 Version 1709 (OS Build 16299.371) Home Edition for one lang x64
7-Zip 18.01 (x64)
Notebook HP 7380er envy dv6, 16GB RAM 1866Mhz Kingston (8Gb x 2)
x64 6.3A09 cpus:8 128T
CMPXCHG MMX SSE RDTSC PAE SSE2 NX SSE3 CMPXCHG16B XSAVE L2AT VIRT_FIRMWARE RDWRFSGSBASE FASTFAIL RDRAND RDTSCP
Intel(R) Core(TM) i7-3610QM CPU @ 2.30GHz (306A9)
Large pages on (manually enabled via gpedit.msc), running tests on cmd.exe as admin
7zG b -mmt1 -slp -md21 - 7zG.exe OK
7z b -mmt1 -slp -md19 -bt 10 >19.txt 2>19_2.txt - OK
7z b -mmt1 -slp -md22 -bt 10 >22.txt 2>22_2.txt - OK
Have you noticed specific success of Intel(R) Core(TM) i7-3xxx & i5-3xxx processors (CPUID of both is (306A9) )?
So the main reason might be in hardware , not OS or other software?
And one more strange thing - the best performance could be achived at 8Mb size dictionary only - not more, not less - is it connected in any way with algorithm implemented?
Last edit: Mikhail Rokhin 2018-04-16
Another problem discovered - however, LP are enabled for Admins & All users
for your info
https://www.chaoticafractals.com/manual/getting-started/enabling-large-page-support-windows
https://www.itechtics.com/easily-enable-group-policy-editor-gpedit-msc-in-windows-10-home-edition/
when launched as ordinary user 7zip fails
7zG b -mmt1 -slp -md1 -bt 10 - OK
...
7zG b -mmt1 -slp -md18 -bt 10 -OK
7zG b -mmt1 -slp -md19 -bt 10 - OK
Failed, when started with the tests first, but after numerous other tests these one are completed successfully - strange behaviour... very strange... smth with rights assign?
---Spontaneous failures ??? ---
7zG b -mmt1 -slp -md20 -bt 10 - 7zG.exe Decoding error / OK after numerous other -mdXX tests
7z b -mmt1 -slp -md20 -bt 10 - OK - Why does GUI Fail ?
7zG b -mmt1 -slp -md21 - 7zG.exe Decoding error / Fails & Closed / OK after numerous other -mdXX tests
7z b -mmt1 -slp -md21 -bt 10 - OK - Why does GUI Fail ?
7z b -mmt1 -slp -md19 -bt 10 >19.txt 2>19_2.txt - OK
7zG b -mmt1 -slp -md19 -bt 10 - OK
7zG b -mmt1 -slp -md23 -bt 10 - OK
7z b -mmt1 -slp -md22 -bt 10 >22.txt 2>22_2.txt - OK
7z b -mmt1 -slp -md22 -bt 10 - OK
7zG b -mmt1 -slp -md23 -bt 10 - OK
Microsoft Windows 10 x64 Home via VirtualBox 5.2.8
1803 [Version 10.0.17134.1]
7zG b -mmt1 -slp -md20 -bt 10 - OK for Adm; OK for User
7z b -mmt1 -slp -md20 -bt 10 - OK for Adm; OK for User
7zG b -mmt1 -slp -md21 - OK for Adm; OK for User
7z b -mmt1 -slp -md21 -bt 10 - OK for Adm; OK for User
7z b -mmt1 -slp -md19 -bt 10 >19.txt 2>19_2.txt - OK for Adm; OK for User
7zG b -mmt1 -slp -md19 -bt 10 - OK for Adm; OK for User
7z b -mmt1 -slp -md22 -bt 10 >22.txt 2>22_2.txt - OK for Adm; OK for User
7zG b -mmt1 -slp -md22 -bt 10 - OK for Adm; OK for User