cppcheck / Discussion / General Discussion: Issues Running MISRA python script

ZMolnar - 2019-02-14

Hi,

I'm running into an error while trying to run the misra.py addon.
I'm trying to run the script on a .dump file, as per the documentation.

Could anybody give some direction what could be wrong?

Please see the details below.

Environment: Command line
Python: python-3.7.2
CppCheck: latest git master

Command with no issues:
F:\WinBuild\Tools\Cppcheck\addons>python cppcheckdata.py syscalls.c.dump
- command executes without errors

Command failing:
F:\WinBuild\Tools\Cppcheck\addons>python misra.py syscalls.c.dump
Traceback (most recent call last):
File "misra.py", line 2191, in <module>
checker.parseDump(item)
File "misra.py", line 2005, in parseDump
data = cppcheckdata.parsedump(dumpfile)
File "F:\WinBuild\Tools\Cppcheck\addons\cppcheckdata.py", line 735, in parsedump
return CppcheckData(filename)
File "F:\WinBuild\Tools\Cppcheck\addons\cppcheckdata.py", line 697, in init
data = ET.parse(filename)
File "C:\Users\zmolnar\AppData\Local\Programs\Python\Python37-32\lib\xml\etree\ElementTree.py", line 1197, in parse
tree.parse(source, parser)
File "C:\Users\zmolnar\AppData\Local\Programs\Python\Python37-32\lib\xml\etree\ElementTree.py", line 598, in parse
self._root = parser._parse_whole(source)
xml.etree.ElementTree.ParseError: no element found: line 578, column 0</module>

all of the python scripts from the addon folder fail the same way. Is there maybe a python package that is missing?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

versat - 2019-02-15

As far as i know there is no additional python package needed. The default python installation should do.

Do you have python 2.7 available and can test if the issue exists with that python version also?
It seems like the scripts are not that well tested under python 3.x, maybe it is some compatibility issue. There already were problems with python 3 that i fixed.

Or can you attach the syscalls.c.dump file or some source code which causes this issue?

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

ZMolnar - 2019-02-19

It will produce the same error with python 2.7.16. I'm attaching bith the dump and the c file, hopefully you can help. I'm just a user when it comes to python.

syscalls.c

syscalls.c.dump

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

ZMolnar - 2019-02-19

Output with python 2.7.16rc1:

python misra.py syscalls.c.dump
Traceback (most recent call last):
File "misra.py", line 2191, in <module>
checker.parseDump(item)
File "misra.py", line 2005, in parseDump
data = cppcheckdata.parsedump(dumpfile)
File "F:\WinBuild\Tools\Cppcheck\addons\cppcheckdata.py", line 735, in parsedump
return CppcheckData(filename)
File "F:\WinBuild\Tools\Cppcheck\addons\cppcheckdata.py", line 697, in init
data = ET.parse(filename)
File "C:\Python27\lib\xml\etree\ElementTree.py", line 1182, in parse
tree.parse(source, parser)
File "C:\Python27\lib\xml\etree\ElementTree.py", line 657, in parse
self._root = parser.close()
File "C:\Python27\lib\xml\etree\ElementTree.py", line 1671, in close
self._raiseerror(v)
File "C:\Python27\lib\xml\etree\ElementTree.py", line 1523, in _raiseerror
raise err
xml.etree.ElementTree.ParseError: no element found: line 578, column 0</module>

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

versat - 2019-02-20

The file syscalls.c.dump you uploaded is incomplete.
It is missing the dump element with all its sub-elements and the closing </dumps> tag is also missing.

If i run ../cppcheck --dump --platform=win64 --suppress=preprocessorErrorDirective syscalls.c i get a similar file but with the dump section included and valid XML.
So the python addons are not really the problem. Maybe we could add errorhandling for invalid XML files, but that does not solve anything.

The main question is why the dump file is incomplete and thus invalid.
If Cppcheck finished successfully it seems like there is a bug in writing the dump files.
I can not reproduce the issue with only the syscalls.c file. Maybe it only happens when you analyze the whole project?

Edit:
As far as i see directly calling cppcheckdata.py does not make any sense. It just defines some classes and functions and does not execute anything. It is only useful if it is used by another python module. So it is no surprise that it does not fail.

Last edit: versat 2019-02-20

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

ZMolnar - 2019-02-22

Hi versat,

Thank you for your help, I've figured out what is the problem:

if the --dump option is combined with the --check-config, then the incomplete dump files are generated.

I've copied all the arguments from the regular cppcheck invocation into the dump command, thus the two options ended up together.

There should be some mutual exclusion detection built into cppcheck for these two options.

The python script now executes, I can start using it.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
- versat - 2019-02-22
  
  --check-config really just checks the include paths and defines and such things. It does not analyze anything or reports real errors. Are you sure that you use this in your regular cppcheck command line?
  
  --check-config and --dump really do not make any sense together. I can imagine that this causes invalid dump files.
  I guess we really should do something to avoid this.
  
  If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

ZMolnar - 2019-02-22

Ok, I have a complete .dump file, the python call doesn't exit with error, BUT:

the python call never finishes either. Using Python 2.7, the attached dump file, the call to misra.py never exists. The command prompt hangs indefinitely.

Any advice on how to figure out what is going on? Is there a verbose mode for misra.py that would show the progress of processing?

newlib_stubs.c.dump

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

ZMolnar - 2019-02-22

If I remove the misrarules.txt rules file, the script goes on, finishes and prints a bunch of stuff on stderr. I will have to see what is wrong in the rules file, it follows the structure as indicated:

Appendix A: Summary of guidelines
Rule 11.4
text
Rule 11.5
text
...

Does the misra.py creates an output file? Or the stderr has to be captured?

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
- versat - 2019-02-22
  
  I have already fixed some encoding problems. Maybe there is still a problem with some special character / encoding. Can you try to look for special characters in your rules file and reduce the content so there are no problems. Then add a special character and see if that is a problem.If you could provide a problematic dummy file that would help.I have already created some dummy files for regression tests. See for example https://github.com/danmar/cppcheck/blob/master/addons/test/misra2012_rules_dummy_utf8.txt
  
  If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
- versat - 2019-02-22
  
  misra.py does not create an output file as far as i know. You have to capture the stderr output.
  
  If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

ZMolnar - 2019-02-22

If I don't supply the rules file, I get output.

But it seems that the script goes and checks every included header file:
[Apguard/System/Src/newlib_stubs.c:61]: (style) misra violation (use --rule-texts=<file> to get proper output) [misra-c2012-3.1]
[STM32F3xx/CMSIS/Device/STM32F3xx/Inc/stm32f398xx.h:5809]: (style) misra violation (use --rule-texts=<file> to get proper output) [misra-c2012-5.4]
[STM32F3xx/CMSIS/Device/STM32F3xx/Inc/stm32f398xx.h:5809]: (style) misra violation (use --rule-texts=<file> to get proper output) [misra-c2012-5.4]
...............</file></file></file>

The dump file was generated with the --suppress=preprocessorErrorDirective (but cppcheck reports that this argument is ignored??).

I was expecting the misra script to check only the configuration the --dump command is supplied with and not to bother with include files that should be excluded.

Here is how the dump files are created (in a makefile):
cppcheck --enable=all --inconclusive --suppress=preprocessorErrorDirective $(DEFS) $(INCLUDES) $(CPPCHECK_CC_INCPATHS) $(VPATH) --output-file=$(BUILD_DIR)/cppcheckresults.txt
cppcheck --enable=all --inconclusive --suppress=preprocessorErrorDirective $(DEFS) $(INCLUDES) $(CPPCHECK_CC_INCPATHS) --dump $(VPATH)
python Tools/CppCheck/addons/misra.py $(ADDON_VPATH) 2>$(BUILD_DIR)/misraresults.txt

Executed from the command line, the behavior is the same.

Any thoughts about this?

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Daniel Marjamäki - 2019-02-23

The dump file was generated with the --suppress=preprocessorErrorDirective (but cppcheck reports that this argument is ignored??).

What exact output do you get?

If it says:

(information) Unmatched suppression: preprocessorErrorDirective

It does not mean that it is ignored. It only means that no matching error was written.

I was expecting the misra script to check only the configuration the --dump command is supplied with and not to bother with include files that should be excluded.

I agree.. The dump file should not contain any excluded include files at all. Can you help out with some minimal example?

a.c:

#ifdef A #include "a.h" #endif ac

a.h:

ah

I generate a dump file with:

cppcheck --dump -D X a.c

The generated dump file does not have any a.h contents ... as expected. The "a.h" does get a "file index" in the dump file but it is not used so that is ok.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

David Ledger - 2020-11-18

I also have the incomplete .dump file xml error. The closing tag sometimes is present, but I have no idea what factors are involved in its presense.

I'm not sure I know how to reproduce with a minimal example. It doesn't happen always... But when it happens it seems keep happening for a while...

Last edit: David Ledger 2020-11-18

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Invoked with approximately the following command:

cppcheck -iF:/project/external/catch2 --enable=all --addon=F:/project/cmake/misra.json --project=F:/project/build/compile_commands.json --inline-suppr --suppress=missingIncludeSystem --suppress=preprocessorErrorDirective --suppress=unmatchedSuppression --cppcheck-build-dir=F:/project/build/cppcheck --std=c++20 --language=c++ --template=gcc --suppressions-list=F:/project/.cppcheckignore

With the following JSON file:

{
    "script": "${CPPCHECK_ADDONS_PATH}/misra.py",
    "args":
    [
        "--rule-texts=${MISRA_PATH}/MISRA-C-2012.txt",
        "--suppress-rules 2.7,12.3"
    ]
}

Sidenote, it fails with or without environmental variables.

With the following source:

source.test.cpp

int source_test()
{
    return 8;
}

The following is the resulting dump file:

<?xml version="1.0"?>
<dumps>
  <platform name="win32A" char_bit="8" short_bit="16" int_bit="32" long_bit="32" long_long_bit="64" pointer_bit="32"/>
  <rawtokens>
    <file index="0" name="F:/project/programs/test/source.test.cpp"/>
    <tok fileIndex="0" linenr="1" column="1" str="int"/>
    <tok fileIndex="0" linenr="1" column="5" str="source_test"/>
    <tok fileIndex="0" linenr="1" column="16" str="("/>
    <tok fileIndex="0" linenr="1" column="17" str=")"/>
    <tok fileIndex="0" linenr="2" column="1" str="{"/>
    <tok fileIndex="0" linenr="3" column="5" str="return"/>
    <tok fileIndex="0" linenr="3" column="12" str="8"/>
    <tok fileIndex="0" linenr="3" column="13" str=";"/>
    <tok fileIndex="0" linenr="4" column="1" str="}"/>
  </rawtokens>
  <suppressions>
    <suppression errorId="missingIncludeSystem" />
    <suppression errorId="preprocessorErrorDirective" />
    <suppression errorId="unmatchedSuppression" />
    <suppression errorId="*" fileName="*external/catch2*" />
  </suppressions>

Note the missing <\dumps> tag.

Last edit: David Ledger 2020-11-19

Georgiy Komarov - 2020-11-19

misra.py is not intended to use with C++. This addon supports only C90 and C99 languages how it's required by MISRA C 2012 document.

Please check this thread about MISRA C++ / AUTOSAR support.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

David Ledger - 2020-11-19

Ah I see. I didnt think that the production of a dump file for a plugin could be affected by the yet to be run plugin. Wouldn't this issue be present for any plugin, regardless of what it is?

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
- Georgiy Komarov - 2020-11-20
  
  Yes, of course, you're right.
  
  This is somewhat strange. I can't reproduce this with Cppcheck (latest version, commit 317a2d0). I use the following command:
  
  cppcheck --dump 1.cpp --inline-suppr \ --suppress=missingIncludeSystem \ --suppress=preprocessorErrorDirective \ --suppress=unmatchedSuppression \ --cppcheck-build-dir=/tmp \ --std=c++20 --language=c++ --template=gcc \ -i/tmp/Catch2/src/catch2/
  
  With the following 1.cpp:
  
  int source_test() { return 8; }
  
  This gives me the correct dump file.
  
  When I create the following addon configuration:
  
  { "script": "/home/jubnzv/Dev/cppcheck/addons/misra.py", "args": [ "--suppress-rules 2.7,12.3" ] }
  
  And use it with the previous command + --addon=test.json, it correctly calls the misra.py and doesn't save the dump file at all.
  
  I assume that the dump file is not deleted completely because of failure in misra.py due to unexpected C++ code as input.
  
  Could you please try to call Cppcheck with your command, but replace --addon option to --dump? This should generate a correct dump.
  
  If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Issues Running MISRA python script

Static source code analysis tool for C and C++ code

Forums

Help

Issues Running MISRA python script

Please see the details below.

Issues Running MISRA python script

Static source code analysis tool for C and C++ code

Forums

Help

Issues Running MISRA python script document.SUBSCRIPTION_OPTIONS = { "thing": "topic", "subscribed": false, "url": "subscribe", "icon": { "css": "fa fa-envelope-o" } };

Please see the details below.

Issues Running MISRA python script