Menu ▾ ▴

#581 buffer overflow detected

open
nobody
None
5
2024-08-28
2024-08-21
Samd42
No

Hi,
In our CI/CD we run 7z to generate an archive. That has worked fine for years. Today, when switching from a docker ubuntu 22.04 container to a docker ubuntu 24.04 container, 7z is crashing.

'7z' 'a' '-bd' '-mx=7' '-ms=on' '-x!boost_1_87_0/ci_boost_common.py' '-x!boost_1_87_0/ci_boost_release.py' 'boost_1_87_0-snapshot.7z' 'boost_1_87_0'

*** buffer overflow detected ***: terminated

That is zipping a directory named 'boost_1_87_0'. The command line parameters have not been modified.

There are two jobs. One with EOL=LF on all the files, and another with EOL=CRLF, to test windows type files. Usually they both succeed. Now, it's the EOL=CRLF set of files which are failing, if that makes any difference, although it shouldn't.

I will retry again in a few weeks with the Ubuntu 24.04.1 point release. Perhaps 24.04.0 is buggy.

https://app.circleci.com/pipelines/github/boostorg/boost/12768/workflows/ed28670a-d1f9-4d0e-b572-aa2b754434bc

Discussion

  • Igor Pavlov

    Igor Pavlov - 2024-08-21

    what version of 7-zip?

    7z

    What size of files and number of files?
    What number of CPU threads , ram size?

     

  • Samd42

    Samd42 - 2024-08-21

    7-Zip 23.01 (x64) : Copyright (c) 1999-2023 Igor Pavlov : 2023-06-20
    64-bit locale=en_US.UTF-8 Threads:15 OPEN_MAX:1048576

    • Around 1GB of files
    • Around 80,000 files
    • CircleCI medium 4GB RAM, 2 vCPUs
     

  • Igor Pavlov

    Igor Pavlov - 2024-08-21

    I shows Threads:15 instead of your 2 vCPUs.
    -mx=7 and 15 cpu threads can require about 4 GB of RAM
    So maybe it was the problem.
    You can reduce the number of working threads with -mmt2 or -mmt4switch.

     

    Last edit: Igor Pavlov 2024-08-21

  • Igor Pavlov

    Igor Pavlov - 2024-08-21

    Correction.
    7-zip doesn't use whole memory, if it knows correct memory size.
    Run 

    7z b

    it can show memory size reported to 7-zip.

     

  • Samd42

    Samd42 - 2024-08-21

    The CI jobs run in a docker container. Both memory and CPU will vary depending on where the container is run. The output above showed my local computer, but the important thing is CircleCI which could be different.
    We reverted back to ubuntu 22.04. In the near future I will try -mmt2 or -mmt4 as you suggest.

     

  • Igor Pavlov

    Igor Pavlov - 2024-08-21

    I want to find the reason of problem.
    So we need 7z b log in that container with failure.
    Note that different linux systems can have different versions of 7-zip. So we need to know exact version of 7-zip of failure case.

     

    Last edit: Igor Pavlov 2024-08-21

  • Samd42

    Samd42 - 2024-08-21

    The container has restricted memory and cpu. However linux diagnostics may show the information from the host OS, not the container, so it's misleading and could be the source of the problem.

    free -h

                   total        used        free      shared  buff/cache   available
Mem:            68Gi        17Gi       2.0Gi        66Mi        49Gi        50Gi
Swap:             0B          0B          0B

    7z

    7-Zip 23.01 (x64) : Copyright (c) 1999-2023 Igor Pavlov : 2023-06-20
 64-bit locale=en_US.UTF-8 Threads:36 OPEN_MAX:65536

    7z b

    ~~~

    7-Zip 23.01 (x64) : Copyright (c) 1999-2023 Igor Pavlov : 2023-06-20
    64-bit locale=en_US.UTF-8 Threads:36 OPEN_MAX:65536

    Compiler: 13.2.0 GCC 13.2.0: SSE2
    Linux : 5.15.0-1057-aws : #63~20.04.1-Ubuntu SMP Mon Mar 25 10:28:36 UTC 2024 : x86_64 : KVMKVMKVM : { : 0.0.0.0.0.0
    PageSize:4KB THP:madvise hwcap:6 hwcap2:2
    Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz (50654)

    1T CPU Freq (MHz): 3371 3373 3379 3344 3352 3329 3326
    18T CPU Freq (MHz): 206% 369 197% 353

    RAM size: 70213 MB, # CPU hardware threads: 36
    RAM usage: 8009 MB, # Benchmark threads: 36

                       Compressing  |                  Decompressing

    Dict Speed Usage R/U Rating | Speed Usage R/U Rating
    KiB/s % MIPS MIPS | KiB/s % MIPS MIPS

    22: 6503 200 3168 6327 | 59460 200 2538 5069
    23: 6079 200 3103 6194 | 58797 200 2547 5087
    24: 6207 200 3341 6674 | 59129 200 2597 5188
    /bin/bash: line 1: 89 Killed 7z b

    ~~~

     

    Last edit: Samd42 2024-08-21

    • Igor Pavlov

      Igor Pavlov - 2024-08-22

      However linux diagnostics may show the information from the host OS, not the container, so it's misleading and could be the source of the problem.

      Yes, 7-zip sees:

      RAM size: 70213 MB
CPU hardware threads: 36

      So by default it tries to use up to 36 cpu threads and up to 80% from RAM size (80% from reported 70213 MB).
      -mmt2 switch will help you, because 2 threads will use only 400 MB of RAM, if -mx=7 switch is used.

      But what about previous container in your tests?
      Why did it work before?
      Is it bad configuration of container?
      What changes do you have?

       

      Last edit: Igor Pavlov 2024-08-22

      • Sam Tansy

        Sam Tansy - 2024-08-22

        -mmt2 switch will help you, because 2 threads will use only 400 MB of RAM, if -mx=7 switch is used.

        When did it change, it used to use much more (~12 * dictionary * threads)?
        Also how does this scheduler work now - for `-mmt=2..3' threads uses 2 threads and ~400MB, for `mmt=4+' uses 5 threads and 1200MB.

         

        • Igor Pavlov

          Igor Pavlov - 2024-08-22

          Each pair of 2 threads work for one lzma2 block:
          One thread searches big matches, and another thread searches small matches and encodes data.

          1 or 2 threads - single block.
          4+threads: more that one block. So it requires additional input and output buffers for each block.

           

          Last edit: Igor Pavlov 2024-08-22

          • Sam Tansy

            Sam Tansy - 2024-08-22

            If OP has more than 1.2GB RAM available in that container they may be able to use 4+ threads.

             

  • Samd42

    Samd42 - 2024-08-22

    -mmt2 switch will help you. OK. Thanks. We'll use that option.

    What changed? Switched from ubuntu22 to ubuntu24. Here is the previous configuration.

    free -h

    total        used        free      shared  buff/cache   available
Mem:            68Gi        17Gi       2.4Gi       117Mi        48Gi        49Gi
Swap:             0B          0B          0B

    7z

    7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,64 bits,36 CPUs Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz (50654),ASM,AES-NI)

    7z b

    7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,64 bits,36 CPUs Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz (50654),ASM,AES-NI)

Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz (50654)
CPU Freq: - - - - - - - - -

RAM size:   70213 MB,  # CPU hardware threads:  36
RAM usage:   7942 MB,  # Benchmark threads:     36

                       Compressing  |                  Decompressing
Dict     Speed Usage    R/U Rating  |      Speed Usage    R/U Rating
         KiB/s     %   MIPS   MIPS  |      KiB/s     %   MIPS   MIPS

22:       4707   200   2293   4579  |      57146   200   2440   4873
23:       4621   200   2357   4709  |      56605   200   2453   4898
24:       4304   200   2318   4628  |      55431   200   2439   4865
/bin/bash: line 1:    89 Killed                  7z b

    After just running some tests locally, it seems that containers show their statistics from the host operating system, rather than the container (which is limited). That presents a difficulty and a puzzle to be able to estimate the available CPU and memory. It's probably worthwhile to investigate more.

     

    • Igor Pavlov

      Igor Pavlov - 2024-08-22

      It has older version of 7-zip: 16.02, but 16.02 version also could requst big memory as I suppose.
      So now I don't know why it worked without problem before.

      Also probably there is some way to configure container to report smaller memory size and number of cpu threads.

       

  • Samd42

    Samd42 - 2024-08-23

    some way to configure container to report smaller memory size

    Not in the usual, expected way. Perhaps by having 7z call docker-specific commands. However I do not know which commands, from inside a container.

     

  • Samd42

    Samd42 - 2024-08-28

    JDK faced these issues before: https://bugs.openjdk.org/browse/JDK-8146115

     

Log in to post a comment.