Windows 10 works incorrectly with Large Memory Pages (2 MB)

2018-03-30
7 hours ago
1 2 3 > >> (Page 1 of 3)
  • Igor Pavlov

    Igor Pavlov - 2018-03-30

    Windows 10 works incorrectly with Large memory pages (2 MB)

    This message contains information about very critical problem in some versions of Windows.
    If you have any contacts with Microsoft, please notify them about this problem. Also ask Microsoft to provide information what versions affetced by that BUG, and

    Now we have the following information about different Windows 10 versions:
    1511 - maybe OK
    1607 - maybe OK
    1703 - ERROR
    1709 - ERROR - including latest version 16299.371 - April 2018 Cumulative
    1803 - maybe OK - but we need more tests
    1809 - maybe OK

    These results are not final. And we need more tests in different versions of Windows, including version 1803 (17133/17134) - Windows 10 Spring Creators Update - Redstone 4.

    Description of problem

    Windows supports small (4 KB) pages and large (2 MB) memory pages.
    Most of programs use only 4 KB pages. And it works correctly. So there is no any problem for most programs.
    But there is Windows API that allows to use 2 MB pages. And 7-Zip uses that API, if special "Large pages mode" option is enabled in 7-Zip settings.
    When 7-Zip uses 2 MB pages, 7-Zip can work incorrectly, or Windows system can crash.

    How to reproduce the problem

    NOTE:
    If you use "Large Pages" now, your system can work incorrectly and you can lose some data.
    DON'T TRY IT, if you don't want to take these RISKS.
    If you use "Large Pages", your system can be in CORRUPTED STATE, and you MUST REBOOT after each experiment to reduce the risk to lose your data.

    1) "Large Pages" feature requires special "LockMemory" privilege.
    So you need to run 7-Zip File Manager with administrator rights at least once and reboot system after that. Then you can use "large pages" without administrator rights in Windows 10.
    But some previous versions of Windows require administrator rights also for programs that just use "large pages".

    2) The commands with "Large Pages" for crash:
    7-Zip 18.01 (x86 or x64):

    7zG b -mmt1 -slp -md21
    7z b -mmt1 -slp -md19 -bt 10 >19.txt 2>19_2.txt
    7z b -mmt1 -slp -md22 -bt 10 >22.txt 2>22_2.txt
    

    If you use Windows 7, you must call these commands with administator rights. So create bat files and run them with administator rights.

    If 7-Zip benchmark works incorrectly with large pages, you can get
    - "Decoder error" message
    - 7-Zip program crash
    - Windows system crash

    When you run these benchmark commands, you can see "LP" string in "Memory usage" values.
    It means that 7-Zip uses "Large Pages".
    If you don't see "LP", then probably you have no rights to use "Large Pages".

    For 7-Zip users:
    If you are ready to test it, please call those benchmark commands and write report here in this forum thread:
    1) Exact windows version. You can call "Run" (Windows+R): "winver" to get version number.
    2) Exact name of CPU
    3) If there are no errors for commands, then show 22.txt
    4) If there is any error/crash, write about that error
    5) REBOOT YOUR SYSTEM after commands to flush possible system corruptions.

    We need to get answers for the following questions:
    1) what Windows versions (and revisions) are affected with that problem?
    2) what Windows versions allow to use "Large Pages" without administrator rights?
    3) what another programs also use "Large Pages"? And are there any reports about problems with these programs?

    Notes

    There are no error reports for Windows 7 still.
    Now we have error reports for Windows 10 only.

    Also there were similar error reports in May-June 2017 for old revisions of Windows 10 (Version 1703). So probably it is not new BUG of latest Windows patches. And Windows 10 contains that BUG for one year at least.
    If you have test systems with all Windows 10 revisions, please check all of them. But check that you see "LP" string in "Memory Usage" string in tests.

    For 7-Zip users

    It's recommended to switch off the option "Use large memory pages" in 7-Zip / Tools / Options / Settings, if you switched it before.

    The confirmation about Windows BUG in another program

    There is another program that also uses large pages - VapourSynth - http://www.vapoursynth.com/
    And they also know about that BUG with large pages.

    There is some code in VapourSynth that works with these bad cases:

    https://github.com/vapoursynth/vapoursynth/blob/master/src/core/vscore.cpp

    static bool isWindowsLargePageBroken() {
        // A Windows bug exists where a VirtualAlloc call immediately after VirtualFree
        // yields a page that has not been zeroed. The returned page is asynchronously
        // zeroed a few milliseconds later, resulting in memory corruption. The same bug
        // allows VirtualFree to return before the page has been unmapped.
    

    Technical description

    7-Zip benchmark allocates 2 (two) buffers of large pages.
    Then probably it still can work OK.
    But 7-Zip or system can work incorrectly after Free() operations.
    It's possible that it still can work correctly, if we allocate only 1 (one) buffer with large pages.

    If VapourSynth's description is correct, it can work so:
    1) 7-Zip allocates large pages with VirtualAlloc(MEM_LARGE_PAGES) - OK.
    2) 7-Zip asks to free large pages with VirtualFree().
    3) Windows puts large pages to some queue for asynchronous filling with zeros. But Windows allows another VirtualAlloc() calls to get these pages.
    4) 7-Zip, some another program or Windows calls VirtualAlloc(),
    (it can be 4 KB pages or 2 MB pages). And Windows sometimes can return same pages, that are still in queue for ZEROing. So we can get situation where two different virtual addresses links to one physical space.
    5) 7-Zip (or Windows) try to use new allocated virtual pages.
    6) Windows asynchronously fills old 2 MB physical pages with zeros.
    7) 7-Zip (or Windows) don't expect that data in allocated pages can be asynchronously changed by another process, so 7-Zip (or Windows) can crash.

    VapourSynth's source code uses some workaround with Sleep() function after VirtualFree(). But actually it can help, if same application tries to allocate memory after VirtualFree(). But what about another applications or system? They also can get these "bad" physical pages after VirtualFree(). So VapourSynth's workaround will not work in that case?

     
    Last edit: Igor Pavlov 6 days ago
    • Enzo Matsumiya

      Enzo Matsumiya - 6 days ago

      Windows 10 Version 1709 (OS Build 16299.371)
      7-Zip 18.01 (x64)

      HP Z600, 2x Intel Xeon E5620 @ 2.4Ghz, 48GB RAM ECC

      Large pages on (manually enabled via gpedit.msc), running tests on cmd.exe as admin

      7zG b -mmt1 -slp -md21 - 7zG.exe crashes
      7z b -mmt1 -slp -md19 -bt 10 >19.txt 2>19_2.txt - OK
      7z b -mmt1 -slp -md22 -bt 10 >22.txt 2>22_2.txt - 7z.exe crashes

      Note: I enabled large pages to use xmr-stak (Monero mining). Since then, I've observed that a few apps (Excel 2016, Telegram, BarTender) were crashing but I didn't correlate those to the large pages stuff.

       
  • Humangenom

    Humangenom - 2018-03-30

    You should also check the microcode version of the cpu. So you can compare the values if the same cpu is working and you get errors from different people.

     
  • theDEXmonster

    theDEXmonster - 2018-03-31

    My system crashed. The error report did not have any information besides: IRQL_NOT_LESS_OR_EQUAL

     
  • Uwe Ruch

    Uwe Ruch - 2018-03-31

    I have submitted the issue to Microsoft via Feedback Hub.

    https://aka.ms/Vj4rew

     
    • bugreporter1694

      bugreporter1694 - 2 days ago

      When I try to access this link via the Feedback Hub I get this error message: "Your account doesn't have access to this feedback".

      I created a new issue on the Feedback Hub in case other readers want to vote. Voting should help the issue get resolved quicker.

      https://aka.ms/Lgf6fv

       
  • Rudi Blom

    Rudi Blom - 2018-03-31

    Tried the three commands, no problem. My device HP Elitebook 745 G2
    . AMD A10 PRO-7350B R6, 10 Compute Cores 4C+6G (630F01)
    . 16GB memory
    . BIOS 1.45
    . Windows 10 Enterprise 64-bit, version 1709 (OS Build 16299 192)
    . 7-Zip 18.01 (x64)

     
    • Rudi Blom

      Rudi Blom - 2018-03-31

      22_2.txt empty, 22.txt:

      7-Zip 18.01 (x64) : Copyright (c) 1999-2018 Igor Pavlov : 2018-01-28

      x64 15.3001 cpus:4 128T
      CMPXCHG MMX SSE RDTSC PAE SSE2 NX SSE3 CMPXCHG16B XSAVE L2AT VIRT_FIRMWARE RDWRFSGSBASE FASTFAIL RDTSCP
      AMD A10 PRO-7350B R6, 10 Compute Cores 4C+6G (630F01)
      CPU Freq: 1361 1032 1015 1032 1163 1024 1091 1092 1074

      RAM size: 15305 MB, # CPU hardware threads: 4
      RAM usage: 57 MB, # Benchmark threads: 1

                         Compressing  |                  Decompressing
      

      Dict Speed Usage R/U Rating | Speed Usage R/U Rating
      KiB/s % MIPS MIPS | KiB/s % MIPS MIPS

      22: 1048 97 1049 1020 | 10978 97 967 937
      22: 1073 99 1053 1044 | 11132 99 960 951
      22: 1109 100 1079 1079 | 11452 99 985 978
      22: 1086 100 1061 1057 | 11054 98 960 944
      22: 1128 99 1107 1098 | 11368 99 981 971
      22: 1060 89 1156 1032 | 9984 91 941 852
      22: 934 84 1088 909 | 12289 96 1091 1049
      22: 954 86 1084 928 | 14010 95 1257 1196
      22: 1091 93 1136 1061 | 10578 95 950 903
      22: 940 92 1000 915 | 11017 98 960 941
      ---------------------------------- | ------------------------------
      Avr: 94 1081 1014 | 97 1005 972
      Tot: 95 1043 993

      Kernel Time = 0.625 = 0% 177032 MCycles
      User Time = 84.296 = 94%
      Process Time = 84.921 = 95% Virtual Memory = 57 MB
      Global Time = 89.026 = 100% Physical Memory = 55 MB

       
      • Igor Pavlov

        Igor Pavlov - 2018-03-31

        Please note that you have no "LP" string.
        So large pages were not used in yor tests.
        You need to run 7-Zip File Manager with administrator rights at least once and reboot system after that.
        If you stll don't see "LP" in tests, then create bat file and run that but file with administrtor rights.

         
        • Rudi Blom

          Rudi Blom - 2018-03-31

          Excuses, you're right. So I did it again with interesting results

          1. right-click on 7zFM.exe and select "run as administrator"
            . check tools -> options -> settings "use large memory pages" marked

          2. after reboot
            T1 hang after 11 sec
            T2 popup "7-Zip GUI has stopped working"; stop and 19_2.txt empty
            T3 popup "7-Zip GUI has stopped working"; stop and 22_2.txt empty

          3. afternew reboot, wait till startup activities completely finished (TaniumClient is a nasty)
            T1 runs OK, after 14m make screenshot and cancel (screenshot attached)
            T2 run OK
            T3 pop-up "7-zip GUI stopped working", 22.txt, 22_2.txt empty

          4. after new reboot, wait till startup activities completely finished
            T3 popup "7-Zip GUI has stopped working"; stop and 22_2.txt empty
            T1 OK, stopped after two passes (about 1 minute)
            T2 OK, 19_2 empty

           
        • Vladimir Presnyakov

          Where "LP" string have to be? (for all 3 commands you give for test)

           
          • Igor Pavlov

            Igor Pavlov - 2018-04-02

            Look attached screnshots and txt files in these thread. And you will see "LP" string.

             
            • Vladimir Presnyakov

              I don't know why, by can't help you with testing because no LP on two my computers.
              (Windows 10, run as administrator and "use large memory pages" is cheked in options of 7zFM.exe)

               
              • Igor Pavlov

                Igor Pavlov - 2018-04-03

                You need to run 7-Zip File Manager with administrator rights at least once and reboot system after that. Then you can use "large pages" without administrator rights in Windows 10.

                If it still doesn't show LP after reboot, write about your version of Windows from winver command.

                 
                • Vladimir Presnyakov

                  You quite right. Now I see LP and start testing.

                   
  • Piotr Biesiada

    Piotr Biesiada - 2018-03-31

    First command gives 7z crash and then Windows crash (MEMORY_MANAGEMENT).
    Second command runs ok for some time, then Ctrl+C and nothing bad.
    Third command gives 7z crash. Windows remain stable.

    WinVer: 1709 - 16299.334
    CPU: intel i7-8700K (not overclocked)
    22.txt: empty

     
  • Walt Huang

    Walt Huang - 2018-03-31

    WinVer: Windows 7 Enterprise X64 with newest hotfix
    CPU: AMD Athlon X4 651k
    test pass. no error was found

     
    Last edit: Walt Huang 2018-03-31
  • Alexey Denisov

    Alexey Denisov - 2018-04-01

    1) Windows 10 Home - 1511 (10586.1177), 7-Zip 18.03 x64, LP enabled
    2) AMD A10-4600M (underclocked to 2.0 GHz). 16Gb RAM.
    3) No errors for all 3 commands

     
    Last edit: Alexey Denisov 2018-04-01
    • Igor Pavlov

      Igor Pavlov - 2018-04-02

      It's interesting that old revision - Windows 10 1511 (10586.1177) works OK.
      So the bug could be introduced in some later revision.
      So we need more results for another revisions.

       
  • Uwe Ruch

    Uwe Ruch - 2018-04-02

    WinVer: Windows 10 Pro RS4 RTM (17133.1 - 1803 - 64Bit).
    CPU: Intel i7-2860QM (including Spectre microcode (MCU: D2) and Meltdown patch).
    22.txt is empty.

    All 3 tests passed. No issues so far.

     
    Last edit: Uwe Ruch 2018-04-02
    • Igor Pavlov

      Igor Pavlov - 2018-04-02

      1) It's new version of Windows.
      2) You tested 8/8 threads. But error for single thread mostly:

      7zG b -mmt1 -slp -md21
      

      3) copy files from 7-Zip folder to another folder, for example, c:\1. Create test.bat file in that c:\1 folder with benchmark commands. And run it from that new folder. So it will have rights to create txt files in that folder.

       
  • Uwe Ruch

    Uwe Ruch - 2018-04-02

    Hi Igor

    I've just tested it in a new folder c:\1 by running a test.bat file.
    All 3 test scenarios passed without any issues. No problems on my system so far.

    Greetings
    Uwe

     
    Last edit: Uwe Ruch 2018-04-02
    • Igor Pavlov

      Igor Pavlov - 2018-04-02

      So it's possible that the BUG was fixed in that new revision of Windows 10 (1803).
      1) You can try to check that GUI benchmark tests with different number of threads and dictionary sizes. But most errors are for single-thread mode.
      2) And if you will install another Windows revisions to that computer, you can test it again.

       
  • Uwe Ruch

    Uwe Ruch - 2018-04-02

    Hi Igor

    I have made a series of tests and there has been not one issue so far. Everything seems to work fine with 1803.

    If I find some time I will try with another Windows 10 version.

    Greets
    Uwe

     
  • 0815

    0815 - 2018-04-02
    Version CodeName MarketingName Build Symptoms LP w/o admin priv.
    1607 RS1 Anniversary Update 10.0.14393.2155 none yes
    1709 RS3 FallCreatorsUpdate 10.0.16299.309 ForceClose, BSOD yes
    1803 RS4 SpringCreatorsUpdate 10.0.17133.1 none yes
    1809 RS5 (PreRelease) 10.0.17618.1000 none yes

    I did run those commands you posted a couple of times.

    On Windows 1709 I can sometimes run the 1st test (7z b -mmt1 -slp -md19 -bt 10 >19.txt 2>19_2.txt) a couple of times without errors.

    The 2nd one (7z b -mmt1 -slp -md22 -bt 10 >22.txt 2>22_2.txt) sometimes works as well when running it after a fresh reboot. But most times it just causes an force close of the 7z.exe process. When that happened the 1st test doesn't work neither anymore unless I reboot.

    One time I even had the case that both tests were passing: https://paste.debian.net/plainh/275e5839 (using a 3rd party build of 7z 1701). But even that build crashes most of the time as well.


    CPU Manufacturer CPU Name Microcode Revision CPUID
    Intel i5-3570K 1F 306A9
     
    Last edit: 0815 2018-04-09
1 2 3 > >> (Page 1 of 3)

Log in to post a comment.