Very large MISRA memory usage regression when using suppression rules

[Richard Smith]

I'm continuing to try and move our build from cppcheck 1.86 up to 1.90+HEAD. I've found another issue.

All of my tests so far have been running on native on my Linux system. Our build system however uses a VM. When I updated the VM to use the newer cppcheck I found that the MISRA check was getting killed.
The problem is that the memory useage of the current development version increases significatly when rule suppressions are used. Our VM only had 1Gig of RAM allocated to it. MISRA eats that up and the OOM killer triggers.

Using memory_profiler on my development machine I've tracked it down to using suppressions. The more rules that are suppressed the more memory it uses. I've only been looking to the while program using mprof run misra.py <options> and not with the @profile annotations memory_profiler provides.

I have been comparing devlopement with 1.86. See below. I have not done any bisection of versions inbetween 1.96 and development. With all the structural changes that occurred lately A/B testing is a PITA.

Heres' some current memory useage numbers. Values are in megs. Source code tree is identical in all runs.

1.86 10 rules active
MEM 780.019531

1.90+HEAD
0 rules
MEM 784.515625

1 rules
MEM 852.695312

2 rules
MEM 1511.367188

4 rules.dat
MEM 1732.757812

10 rules
MEM 1778.050781

You can see that 1.86 with 10 rules has the same memory uses as no rules when using developement. When rules suppressions are used the memory usage increses by a large margin.

On a different but related note. If you watch misra.py's memory usage over time even 1.86 seems to be excessive (780 megs? Thats really large) as the program runs the memory usage steadly increases. Perhaps the new objects that are created as it checks each new file are not released when it moves to a new file?

In our setup we feed misra.py what is esentially misra.py *.dump. The result is 111 files provided on the command line.

I can't really put up my code base for test. Are there any open source projects using the MISRA checker that can try to duplicate my findings?

[Daniel Marjamäki]

spontanously .. it sounds strange that suppressions requre lots of memory.

[Daniel Marjamäki]

I can't really put up my code base for test. Are there any open source projects using the MISRA checker that can try to duplicate my findings?

I don't know. :-(

[versat]

Which Python version are you using?
Does it make any difference using Python 2 compared to Python 3?

[Richard Smith]

The regression first appears in version 1.89 although I was unable to test version 1.88 because it just thows an execption when I try to run it.

Which Python version are you using?
Does it make any difference using Python 2 compared to Python 3?

I did that testing with python2.7. I didn't know python3 was fully supported. Do the unit tests now work when using python3?

Python2 vs 3 does make a differenece. Although the regression still exists.
With python3 HEAD only uses a 728.906250 vs 1.87 using 305.074219
with the same setup.

[versat]

Yes Python 3 is (or really should be) fully supported for the addon scripts. Otherwise it is a bug.
The unit tests for the addons are currently only executed with Python 2 by Travis. But i made a Pull Request to add execution with Python 3 also: https://github.com/danmar/cppcheck/pull/2185
Locally the addon unit tests work with Python 3.
There are other tests for the addons that are already executed with Python 2 and Python 3 by Travis.

Regarding your issue i do not really have an idea how to solve that, but i have not that deep of an insight into the addon scripts.
How does your command line and especially the suppressions look like? Can you paste the suppressions with censored file names/paths or so? Maybe that helps to reproduce or find the issue.

[Richard Smith]

Locally the addon unit tests work with Python 3.

Doesn't work for me. Should probablly take this into another thread.

rsmith@rsmith-xps13:/home/src/cppcheck.git$ python3 -m pytest addons/test/test-misra.py
========================================================================= test session starts =========================================================================
platform linux -- Python 3.6.8, pytest-3.3.2, py-1.5.2, pluggy-0.6.0
rootdir: /home/src/cppcheck.git, inifile:
collected 9 items

addons/test/test-misra.py EEEEEEEEF [100%]

=============================================================================== ERRORS ================================================================================
________ ERROR at setup of testloadRuleTexts_structure __________

@pytest.fixture(scope="function")
def checker():

  from addons.misra import MisraChecker, MisraSettings, get_args

addons/test/test-misra.py:26:

---

from __future__ import print_function

import cppcheckdata
E ModuleNotFoundError: No module named 'cppcheckdata'

addons/misra.py:18: ModuleNotFoundError

[Richard Smith]

Also I don't think the unit tests for misra.py in python2 are actually gettting run because they currently generate errors.

rsmith@rsmith-xps13:/home/src/cppcheck.git$ python -m pytest addons/test/test-misra.py
========================================================================= test session starts =========================================================================
platform linux2 -- Python 2.7.15+, pytest-3.3.2, py-1.5.2, pluggy-0.6.0
rootdir: /home/src/cppcheck.git, inifile:
collected 9 items

addons/test/test-misra.py .....FFF. [100%]

============================================================================== FAILURES ===============================================================================
____________ test_json_out ______________

checker = <addons.misra.misrachecker instance="" 0x7f197bf103f8="" at="">, capsys = <_pytest.capture.CaptureFixture instance at 0x7f197bf10d40></addons.misra.misrachecker>

def test_json_out(checker, capsys):
    sys.argv.append("--cli")
    checker.loadRuleTexts("./addons/test/misra/misra_rules_dummy.txt")

  checker.parseDump("./addons/test/misra/misra-test.c.dump")

addons/test/test-misra.py:74:

---

addons/misra.py:2321: in parseDump
data = cppcheckdata.parsedump(dumpfile)
addons/cppcheckdata.py:800: in parsedump
return CppcheckData(filename)
addons/cppcheckdata.py:757: in init
tok = Token(node)

---

self = <addons.cppcheckdata.token instance="" at="" 0x7f197b4e27e8="">, element = <element 0x7f197bf04b50="" at="" 'tok'=""></element></addons.cppcheckdata.token>

def __init__(self, element):
    self.Id = element.get('id')
    self.str = element.get('str')
    self.next = None
    self.previous = None
    self.scopeId = element.get('scope')
    self.scope = None
    type = element.get('type')
    if type == 'name':
        self.isName = True
        if element.get('isUnsigned'):
            self.isUnsigned = True
        if element.get('isSigned'):
            self.isSigned = True
    elif type == 'number':
        self.isNumber = True
        if element.get('isInt'):
            self.isInt = True
        elif element.get('isFloat'):
            self.isFloat = True
    elif type == 'string':
        self.isString = True
        self.strlen = int(element.get('strlen'))
    elif type == 'char':
        self.isChar = True
    elif type == 'op':
        self.isOp = True
        if element.get('isArithmeticalOp'):
            self.isArithmeticalOp = True
        elif element.get('isAssignmentOp'):
            self.isAssignmentOp = True
        elif element.get('isComparisonOp'):
            self.isComparisonOp = True
        elif element.get('isLogicalOp'):
            self.isLogicalOp = True
    if element.get('isExpandedMacro'):
        self.isExpandedMacro = True
    self.linkId = element.get('link')
    self.link = None
    if element.get('varId'):
        self.varId = int(element.get('varId'))
    self.variableId = element.get('variable')
    self.variable = None
    self.functionId = element.get('function')
    self.function = None
    self.valuesId = element.get('values')
    self.values = None
    if element.get('valueType-type'):
        self.valueType = ValueType(element)
    else:
        self.valueType = None
    self.typeScopeId = element.get('type-scope')
    self.typeScope = None
    self.astParentId = element.get('astParent')
    self.astParent = None
    self.astOperand1Id = element.get('astOperand1')
    self.astOperand1 = None
    self.astOperand2Id = element.get('astOperand2')
    self.astOperand2 = None
    self.file = element.get('file')
    self.linenr = int(element.get('linenr'))

  self.column = int(element.get('column'))

E TypeError: int() argument must be a string or a number, not 'NoneType'

addons/cppcheckdata.py:254: TypeError

---

[Richard Smith]

Regarding your issue i do not really have an idea how to solve that, but i have not that deep of an insight into the addon scripts.
How does your command line and especially the suppressions look like? Can you paste the suppressions with censored file names/paths or so? Maybe that helps to reproduce or find the issue.

I have tested it with various arguments. The problem occurs either by adding global suppressions using --suppress-rules or by incorporating them into the .dump file by using --suppressions-list= from cppcheck. This would be expected since the acutall implementation of the rules is the same in both cases. Only the loading method is different. All the other arguments I've tested to not have any effect.

I've narrowed re-createing the problem down to the number of files included on the commnd line. There are 2 issues:

First the ammount of memory appear to grow as you check more files ie:

python misra.py --suppress-rules = "5.4,11.4,13.1" file1.c.dump file2.c.dump file3.c.dump

uses more ram than:

python misra.py --suppress-rules = "5.4,11.4,13.1" file1.c.dump

where you can replace 'file1.c.dump' in the 2nd example with any of the other files. This happens in both 1.90 and 1.87

2nd is that the amount by which that memory increases is much larger if you do that test using 1.90 vs 1.87

The test files I'm using are the ones in our code base that generate the largest dump files. The biggest (file1.c.dump) is 7Megs and the other 2 are 5 Megs and 4 Megs.

[Richard Smith]

oops. That should be

python misra.py --suppress-rules "5.4,11.4,13.1" file1.c.dump file2.c.dump file3.c.dump

[versat]

According to the Travis log files and my local tests it works.
The command lines are slightly different from the ones you used:

python -m pytest addons/test/test-*.py
PYTHONPATH=./addons python3 -m pytest addons/test/test-*.py

[Richard Smith]

That command line came from the comment in the ./addons/test/test-misra.py file
That file needs updating with the correct usage.

Using your updated command line I still have failures but now the failure is the same for both python 2 and python 3. :) I've pasted the 1st failure below there are multiple failures with the same error.

rsmith@rsmith-xps13:/home/src/cppcheck.git$ PYTHONPATH=./addons python3 -m pytest addons/test/test-*.py
========================================================================== test session starts ==========================================================================
platform linux -- Python 3.6.8, pytest-3.3.2, py-1.5.2, pluggy-0.6.0
rootdir: /home/src/cppcheck.git, inifile:
collected 15 items                                                                                                                                                      

addons/test/test-cert.py .                                                                                                                                        [  6%]
addons/test/test-misra.py .....FFF.                                                                                                                               [ 66%]
addons/test/test-y2038.py FFFF.                                                                                                                                   [100%]

=============================================================================== FAILURES ================================================================================
_____________________________________________________________________________ test_json_out _____________________________________________________________________________

checker = <addons.misra.MisraChecker object at 0x7f68ea0fae48>, capsys = <_pytest.capture.CaptureFixture object at 0x7f68ea0fa240>

    def test_json_out(checker, capsys):
        sys.argv.append("--cli")
        checker.loadRuleTexts("./addons/test/misra/misra_rules_dummy.txt")
>       checker.parseDump("./addons/test/misra/misra-test.c.dump")

addons/test/test-misra.py:74: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
addons/misra.py:2321: in parseDump
    data = cppcheckdata.parsedump(dumpfile)
addons/cppcheckdata.py:800: in parsedump
    return CppcheckData(filename)
addons/cppcheckdata.py:757: in __init__
    tok = Token(node)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <cppcheckdata.Token object at 0x7f68e9f34390>, element = <Element 'tok' at 0x7f68ea0fec28>

    def __init__(self, element):
        self.Id = element.get('id')
        self.str = element.get('str')
        self.next = None
        self.previous = None
        self.scopeId = element.get('scope')
        self.scope = None
        type = element.get('type')
        if type == 'name':
            self.isName = True
            if element.get('isUnsigned'):
                self.isUnsigned = True
            if element.get('isSigned'):
                self.isSigned = True
        elif type == 'number':
            self.isNumber = True
            if element.get('isInt'):
                self.isInt = True
            elif element.get('isFloat'):
                self.isFloat = True
        elif type == 'string':
            self.isString = True
            self.strlen = int(element.get('strlen'))
        elif type == 'char':
            self.isChar = True
        elif type == 'op':
            self.isOp = True
            if element.get('isArithmeticalOp'):
                self.isArithmeticalOp = True
            elif element.get('isAssignmentOp'):
                self.isAssignmentOp = True
            elif element.get('isComparisonOp'):
                self.isComparisonOp = True
            elif element.get('isLogicalOp'):
                self.isLogicalOp = True
        if element.get('isExpandedMacro'):
            self.isExpandedMacro = True
        self.linkId = element.get('link')
        self.link = None
        if element.get('varId'):
            self.varId = int(element.get('varId'))
        self.variableId = element.get('variable')
        self.variable = None
        self.functionId = element.get('function')
        self.function = None
        self.valuesId = element.get('values')
        self.values = None
        if element.get('valueType-type'):
            self.valueType = ValueType(element)
        else:
            self.valueType = None
        self.typeScopeId = element.get('type-scope')
        self.typeScope = None
        self.astParentId = element.get('astParent')
        self.astParent = None
        self.astOperand1Id = element.get('astOperand1')
        self.astOperand1 = None
        self.astOperand2Id = element.get('astOperand2')
        self.astOperand2 = None
        self.file = element.get('file')
        self.linenr = int(element.get('linenr'))
>       self.column = int(element.get('column'))
E       TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'

addons/cppcheckdata.py:254: TypeError

[versat]

Here is the output when executing it locally on my debian system:

s@debian9:~/cppcheck$ python -m pytest addons/test/test-misra.py
======================================================================== test session starts =========================================================================
platform linux2 -- Python 2.7.13, pytest-4.6.4, py-1.8.0, pluggy-0.12.0
rootdir: /home/s/cppcheck
collected 9 items                                                                                                                                                    

addons/test/test-misra.py .........                                                                                                                            [100%]

====================================================================== 9 passed in 1.29 seconds ======================================================================

I remember that someone reported problems with the wrong/old pytest version.
Travis explicitly uses pytest 4.6.4 with Python 2. See commit https://github.com/danmar/cppcheck/commit/413a5a486571c04197c966fefe780fc54aeedc9c#diff-354f30a63fb0907d4ad57269548329e3
The output with Python 3 on my local system is this:

s@debian9:~/cppcheck$ PYTHONPATH=./addons python3 -m pytest addons/test/test-misra.py
=========================================================================== test session starts ===========================================================================
platform linux -- Python 3.5.3, pytest-5.1.1, py-1.8.0, pluggy-0.12.0
rootdir: /home/s/cppcheck
collected 9 items                                                                                                                                                         

addons/test/test-misra.py .........                                                                                                                                 [100%]

============================================================================ 9 passed in 0.69s ============================================================================

[versat]

I have updated the documentation about running the tests:
https://github.com/danmar/cppcheck/commit/dc1cdd2b761e3f521269d44ead39d1a5d7488ff2
I hope it helps, if anything is unclear or could still be enhanced we can further improve it.

[Richard Smith]

pytest-3.3.2 is what is in Ubutu LTS 18 if the tests require a minimum version of pytest then they should check for that. I susggest that rules be added to the build system that check for minimim versions of the packages required and warnings genearated if they are not satisfied. Not matching the tests probally should not be an error but there should be a warning at minimum.

Doing the units tests shoul also have its own build target so you can do
make test and have check dependencies and then run all the units tests.

I have figued out my problem.

Updating pytest did not fix it so I looked into it a bit more and noticed that the eception occurs in parse_dump() and I realized that I have seen this error before. The error occurs when there is a mismatch between the version of cppcheck used to create the dump file and the version of misra.py run.

The problem is due to the following:

rsmith@rsmith-xps13:/home/src/cppcheck.git$ ./cppcheck --version 
Cppcheck 1.87
rsmith@rsmith-xps13:/home/src/cppcheck.git$ ./build/bin/cppcheck --version
Cppcheck 1.90 dev

The units tests use ./cppcheck to generate the .dump files and in my case that is from an older cppcheck.

If I change dump_create() to use ./build/bin/cppcheck instead of ./cppcheck

This would be another arguement for a make test so you can standarize the enviroment prior to running the tests.

The problem occurs on my system because I have small little helper scripts for building cppcheck due to all the changes across the versions. Some version need SRCDIR, some need, MATCHCOMPILER, some need FILESDIR, etc. For cppcheck < 1.90 I was using the make based build but for 1.90 I switched to using CMake. Aparently the CMake build does not update ./cppcheck and so I had a version mismatch.

[Richard Smith]

Opps. Editing error.

If I change dump_create() to use ./build/bin/cppcheck instead of ./cppcheck

Shoud be:

If I change dump_create() to use ./build/bin/cppcheck instead of ./cppcheck it works and the unit tests complete without problems.

[Georgiy Komarov]

I fixed it. Here is PR with issue description: https://github.com/danmar/cppcheck/pull/2321

@whoopsmith could you please check this version?