Next test on i5-6400, stock clocks, 16GB DDR3 1600 CL9 (2x8 GB), Win10 x64, drive c: is SSD, d: is HDD (some seagate 7200 rpm 2TB). Same testfiles as above on drive d:, output dir on c:.
Synthetic benchmarks run 3 times to check if results stay the same.
Last file was compressed to same solid block with previous file. So it's correct, that 7-zip shows nly pack size of previous file, where new solid block starts.
Some notes about speed.
PPMD lines are slow in your i5-6400, but PPMD lines are faster in some another intel i5/i7 systems. Now I don't know why it's slow sometimes. Maybe some systems have microcode updates that make CPU slow for PPMD, and some another processors do not have such updates.
Last edit: Igor Pavlov 2020-02-09
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I compared your previous results of i5-6400 posted at 2019-September with results of another i5-6500 system.
PPMD has largest difference. But Deflate:x1 is also affected.
So there was some speed downgrade in September already.
Did you install updates before September?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Yes, but this was already a year ago, i then updated BIOS with C6, after that there was a windows update available which did load firmware at boot to CC. And before C6 there was C2 installed in BIOS, and before that an even older version.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I looked Effec % column for PPMD lines (single-thread) in results posted here at forum for intel cpus for 7-Zip 19.02. And only i5-8350U rusult also shows low speed similar to your rusults.
It means that most of systems don't have these low speed updates.
It can be interesting to know what exact update reduces the performance of intel cpus.
Last edit: Igor Pavlov 2020-02-10
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
HITCHER:
I've compared your i5-6400 rusults of 7-Zip 19.02 from September and latest February rusults.
And there is big difference in some tests.
The biggest difference in BCJ test.
I've checked assembler for that BCJ code, and I suppose it's slow because of latest "jump-conditional-code-erratum" problem.
So now we have at least two groups of slow down:
1) slow PPMD - with some old Intel update. Probably it's CVE-2017-5715 (Spectre Variant 2).
2) slow BCJ in x64 - latest Intel jump-conditional-code-erratum
The condition for slow BCJ - is crossing 32-bytes range in jcc instruction is related to x64 version of 7-Zip, but 32-bit version of 7-Zip doesn't cross 32-bytes range for critical loop.
So 32-bit benchmark of 7-Zip must show better numbers in BCJ lines in your i5-6400. You can check it.
Last edit: Igor Pavlov 2020-02-12
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Right now, i cannot check anything on this machine, because it it is out of order.
What happened is, i downgraded UEFI to very old version from 2016, before the fixes for intel security vunerabilties meltdown and spectre came out.
This went well, and i could boot windows normally and run some benchmarks,
but the latest windows update for loading intel microcodes at boot time was still installed, and did load µC CC.
So i deinstalled this update and rebootet again.
Then it happened during start, windows itself did try to update the bios to newer version, which failed late in the progress somehow. Now computer cannot boot from anything (USB-Stick, DVD, HDD, ...) anymore and locks up after showing boot screen.
I will try to fix it, i had made a backup image file from the UEFI from previous version about 1 year ago, but it may take some days.
Last edit: HITCHER 2020-02-12
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
So does Windows update BIOS without user approval?
I try to figure out the total status with updates for all Intel CPUs in use. Do users really have all these updates installed in their computers?
Statistics with 7-Zip 19.02 in September doesn't show PPMd speed degradation for most of systems.
It can mean that Windows doesn't patch by default for all.
And manual BIOS update is required. And probably most of users don't refresh BIOS and have no update.
And same question about latest Intel's jump-conditional-code-erratum. Does windows can install update for that , or manual BIOS update is required?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I did that last update to µC D6 previously myself. Windows does only update skylake to µC CC by now by loading it at boot-time. But Windows also distributes full UEFI updates for some machines from big companies like dell, lenovo, acer, etc. to fix the meltdown & spectre issues, and they could be a problem if gone wrong.
I managed fo fix this corrupt bios problem today, with my old backup file of the BIOS and an external SPI ROM flash programmer.
There are the benchmarks i did before the problem b-cc.txt with µC CC.
And now as i have older µC C6 running, did benchmarks b-c6.txt and b-32bit.txt.
Now I suppose that PPMD slowdown is because of "Spectre Variant 4 (CVE-2018-3639: "Speculative Store Bypass (SSB)")".
And C6 update on skylake already contains fix for Variant 4.
But Microsoft doesn't enable CVE-2018-3639 by default in Windows 10 as I suppose. So most Windows users do not see any slowdown in ppmd and in another places.
Did you enable CVE-2018-3639 protection manually in your Windows with some tool or with registry updating?
We can try to estimate exact numbers of slowdown in two modes:
1) CVE-2018-3639 mitigations enabled in Windows
2) CVE-2018-3639 mitigations disabled in Windows
And microcode CC is OK for such tests.
Last edit: Igor Pavlov 2020-02-16
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Thanks for that testing.
I plan to change PPMd decoding code to increase the speed on patched systems.
It was important to see that CVE-2018-3639 patch affects so strongly also another codes including LZMA:x1, Deflate and BZip2. So I'll look that code too.
So what exact total statistics about patches we have now for all working computers?
Windows does not enable CVE-2018-3639 patch by default?
But what about Linux, including servers?
Another question about performance drops with latest jump conditional code erratum. Will that new microcode and JCC patch be enabled by default in Linux and in future Windows updates?
Last edit: Igor Pavlov 2020-02-17
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I not know exact statistics, but i assume on servers with important data that needs to be secure and also with lots of users, all security fixes / mitigations should be enabled.
It is far more important to keep system secure, than having optimal speed.
There was even the recommendation by Open BSD and Linux to disable Hyperthreading for higher security, which has huge performance impact. https://www.theregister.co.uk/2019/10/29/intel_disable_hyper_threading_linux_kernel_maintainer/
Hopefully Intel fixes most bugs in their future processors, so mitigations and code changes are then not needed anymore. And if code runs still good on other hardware like AMD Ryzen, it is probably not worth doing optimizations for then old and faulty intel chips just for little speed gains.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
So does Windows update BIOS without user approval?
Yes, they started implementing it in W10
- on some motherboards it's not enabled
- on some it's upon user request(device manager->"Firmware" shows as a device->update driver)
- on some(main oems, popular boards) it's automated like any windows update
Last edit: Nicola Bressanin 2020-02-16
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Is it possible to use multithreading for AES256CBC:1 encryption as shown in benchmark results, where KiB/s for 4 threads is shown as roughly 4x of 1 thread result, when creating 7-zip archive?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
1900 vanilla vs 1900 7za vs 2000. Do note that 7za v1900 is compiled by me with MSVC 2019, /O2, LTCG and MT on (NEW_COMPILER=1 MY_STATIC_LINK=1 CPU=AMD64).
I have a 980 MB 8bit Tiff (no LZW). With 4 core cpu I cant compress it, IF I choose 4 cores for the job. Intel 2500k/4.6 and 8GB superfast RAM.
With 3 of 4 cores it works, but is very slow. But 7z do it. Result is a ~175 MB file in 6 minutes and ave. cpu load in task-manager is ~37%. ~2.5 MB/s speed...
With 4 cores, the progress bar "flying" in 1 second to 50% and... 7z do nothing more. Also for the next 20 minutes... But I can abort the operation in the GUI. UI is not frozen.
I have also a 263 MB Tif. It works with 4 cores. With ~2MB/s speed...
Settings are 7z, max, lzma2, 64mb, 192, 4GB and f=bcj2:d15M (GUI, after kontext menu point)
p.s.1:
Winrar5 with the 980MB Tif: 225MB in ~60s.
p.s.2:
mem req info in the gui for 3 cores, 680MB. For 4 cores, ~2.7 GB? Wow...
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Next test on i5-6400, stock clocks, 16GB DDR3 1600 CL9 (2x8 GB), Win10 x64, drive c: is SSD, d: is HDD (some seagate 7200 rpm 2TB). Same testfiles as above on drive d:, output dir on c:.
Synthetic benchmarks run 3 times to check if results stay the same.
Last file was compressed to same solid block with previous file. So it's correct, that 7-zip shows nly pack size of previous file, where new solid block starts.
Some notes about speed.
PPMD lines are slow in your i5-6400, but PPMD lines are faster in some another intel i5/i7 systems. Now I don't know why it's slow sometimes. Maybe some systems have microcode updates that make CPU slow for PPMD, and some another processors do not have such updates.
Last edit: Igor Pavlov 2020-02-09
This may be possible reason. I recently updated bios to new firmeware d6 for skylake:
"SKL-H/S/E3 N0/R0/S0 6-5e-3/36 000000d4->000000d6 Core Gen6"
https://github.com/intel/Intel-Linux-Processor-Microcode-Data-Files/blob/master/releasenote
I compared your previous results of i5-6400 posted at 2019-September with results of another i5-6500 system.
PPMD has largest difference. But Deflate:x1 is also affected.
So there was some speed downgrade in September already.
Did you install updates before September?
Yes, but this was already a year ago, i then updated BIOS with C6, after that there was a windows update available which did load firmware at boot to CC. And before C6 there was C2 installed in BIOS, and before that an even older version.
I looked
Effec %column for PPMD lines (single-thread) in results posted here at forum for intel cpus for 7-Zip 19.02. And only i5-8350U rusult also shows low speed similar to your rusults.It means that most of systems don't have these low speed updates.
It can be interesting to know what exact update reduces the performance of intel cpus.
Last edit: Igor Pavlov 2020-02-10
If it was because of the last update, there are some articles about it (this one in german):
https://www.heise.de/security/meldung/Intel-fixt-Sicherheitsluecken-und-enthuellt-nebenbei-eine-neue-ZombieLoad-Variante-4584543.html
On this site is also the link to new cpu microcode firmware D6 for skylake.
https://www.intel.com/content/dam/support/us/en/documents/processors/mitigations-jump-conditional-code-erratum.pdf
Of course the older OS security mitigations and cpu microcode updates did cost some performance too.
Slow PPMD in your i5 was already in September.
If we want to find exact update that reduces the speed, we can try all old BIOS versions and run the benchmark in each of them.
Last edit: Igor Pavlov 2020-02-11
HITCHER:
I've compared your i5-6400 rusults of 7-Zip 19.02 from September and latest February rusults.
And there is big difference in some tests.
The biggest difference in BCJ test.
I've checked assembler for that BCJ code, and I suppose it's slow because of latest "jump-conditional-code-erratum" problem.
So now we have at least two groups of slow down:
1) slow PPMD - with some old Intel update. Probably it's CVE-2017-5715 (Spectre Variant 2).
2) slow BCJ in x64 - latest Intel jump-conditional-code-erratum
The condition for slow BCJ - is crossing 32-bytes range in jcc instruction is related to x64 version of 7-Zip, but 32-bit version of 7-Zip doesn't cross 32-bytes range for critical loop.
So 32-bit benchmark of 7-Zip must show better numbers in BCJ lines in your i5-6400. You can check it.
Last edit: Igor Pavlov 2020-02-12
Right now, i cannot check anything on this machine, because it it is out of order.
What happened is, i downgraded UEFI to very old version from 2016, before the fixes for intel security vunerabilties meltdown and spectre came out.
This went well, and i could boot windows normally and run some benchmarks,
but the latest windows update for loading intel microcodes at boot time was still installed, and did load µC CC.
So i deinstalled this update and rebootet again.
Then it happened during start, windows itself did try to update the bios to newer version, which failed late in the progress somehow. Now computer cannot boot from anything (USB-Stick, DVD, HDD, ...) anymore and locks up after showing boot screen.
I will try to fix it, i had made a backup image file from the UEFI from previous version about 1 year ago, but it may take some days.
Last edit: HITCHER 2020-02-12
I wish you to fix these BIOS problems.
So does Windows update BIOS without user approval?
I try to figure out the total status with updates for all Intel CPUs in use. Do users really have all these updates installed in their computers?
Statistics with 7-Zip 19.02 in September doesn't show PPMd speed degradation for most of systems.
It can mean that Windows doesn't patch by default for all.
And manual BIOS update is required. And probably most of users don't refresh BIOS and have no update.
And same question about latest Intel's jump-conditional-code-erratum. Does windows can install update for that , or manual BIOS update is required?
I did that last update to µC D6 previously myself. Windows does only update skylake to µC CC by now by loading it at boot-time. But Windows also distributes full UEFI updates for some machines from big companies like dell, lenovo, acer, etc. to fix the meltdown & spectre issues, and they could be a problem if gone wrong.
I managed fo fix this corrupt bios problem today, with my old backup file of the BIOS and an external SPI ROM flash programmer.
There are the benchmarks i did before the problem b-cc.txt with µC CC.
And now as i have older µC C6 running, did benchmarks b-c6.txt and b-32bit.txt.
Last edit: HITCHER 2020-02-15
Now I suppose that PPMD slowdown is because of "Spectre Variant 4 (CVE-2018-3639: "Speculative Store Bypass (SSB)")".
And
C6update on skylake already contains fix for Variant 4.But Microsoft doesn't enable CVE-2018-3639 by default in Windows 10 as I suppose. So most Windows users do not see any slowdown in ppmd and in another places.
Did you enable CVE-2018-3639 protection manually in your Windows with some tool or with registry updating?
We can try to estimate exact numbers of slowdown in two modes:
1) CVE-2018-3639 mitigations enabled in Windows
2) CVE-2018-3639 mitigations disabled in Windows
And microcode CC is OK for such tests.
Last edit: Igor Pavlov 2020-02-16
Yes, mitigations for CVE-2018-3639 were enabled in all previous tests. I patched the system always with everything new available ;-).
So the benchmarks in this posting are with µC CC loaded.
Last edit: HITCHER 2020-02-16
Thanks for that testing.
I plan to change PPMd decoding code to increase the speed on patched systems.
It was important to see that CVE-2018-3639 patch affects so strongly also another codes including LZMA:x1, Deflate and BZip2. So I'll look that code too.
So what exact total statistics about patches we have now for all working computers?
Windows does not enable CVE-2018-3639 patch by default?
But what about Linux, including servers?
Another question about performance drops with latest jump conditional code erratum. Will that new microcode and JCC patch be enabled by default in Linux and in future Windows updates?
Last edit: Igor Pavlov 2020-02-17
I not know exact statistics, but i assume on servers with important data that needs to be secure and also with lots of users, all security fixes / mitigations should be enabled.
It is far more important to keep system secure, than having optimal speed.
There was even the recommendation by Open BSD and Linux to disable Hyperthreading for higher security, which has huge performance impact.
https://www.theregister.co.uk/2019/10/29/intel_disable_hyper_threading_linux_kernel_maintainer/
Hopefully Intel fixes most bugs in their future processors, so mitigations and code changes are then not needed anymore. And if code runs still good on other hardware like AMD Ryzen, it is probably not worth doing optimizations for then old and faulty intel chips just for little speed gains.
Yes, they started implementing it in W10
- on some motherboards it's not enabled
- on some it's upon user request(device manager->"Firmware" shows as a device->update driver)
- on some(main oems, popular boards) it's automated like any windows update
Last edit: Nicola Bressanin 2020-02-16
Is it possible to use multithreading for AES256CBC:1 encryption as shown in benchmark results, where KiB/s for 4 threads is shown as roughly 4x of 1 thread result, when creating 7-zip archive?
It was not implemeted. Multithreading was implemented in another level in code for 7z format.
You can use Multithreading AES with ZIP format in 7-Zip.
Attached results of all tests. Compression test was done on 1GiB driver installation folder.
Compression test consists of my "My Documents" folder that has accumulated over the years.
Here is the benchmark results for a 3900X with 32GB of ram.
adding files to an existing 7z archive isn't compressing them with the same rate/settings as the archive :/
1900 vanilla vs 1900 7za vs 2000. Do note that 7za v1900 is compiled by me with MSVC 2019, /O2, LTCG and MT on (
NEW_COMPILER=1 MY_STATIC_LINK=1 CPU=AMD64).Last edit: XhmikosR 2020-02-27
Hi (sorry for my english)
I have a 980 MB 8bit Tiff (no LZW). With 4 core cpu I cant compress it, IF I choose 4 cores for the job. Intel 2500k/4.6 and 8GB superfast RAM.
With 3 of 4 cores it works, but is very slow. But 7z do it. Result is a ~175 MB file in 6 minutes and ave. cpu load in task-manager is ~37%. ~2.5 MB/s speed...
With 4 cores, the progress bar "flying" in 1 second to 50% and... 7z do nothing more. Also for the next 20 minutes... But I can abort the operation in the GUI. UI is not frozen.
I have also a 263 MB Tif. It works with 4 cores. With ~2MB/s speed...
Settings are 7z, max, lzma2, 64mb, 192, 4GB and f=bcj2:d15M (GUI, after kontext menu point)
p.s.1:
Winrar5 with the 980MB Tif: 225MB in ~60s.
p.s.2:
mem req info in the gui for 3 cores, 680MB. For 4 cores, ~2.7 GB? Wow...