Informational Exit Codes

[Nathan]

Hi,

I was perusing the source on github looking into the exit codes CppCheck provides when something goes wrong (there's nothing in the docs about exit codes of CppCheck, the only mention of the word exit is in relation to the --help parameter of the HTML report generator plugin)

It would be nice if CppCheck provided different exit codes for different errors.. This would help me gate automated build processes with CppCheck.

Currently CppCheck exits 0 when there are problems with the scanned code that aren't syntax errors.. but how about the following?
0 for success (no code issues or CppCheck issues)
MAX_INT for general failure of CppCheck
exit code > 0 for number of warnings/issues found by CppCheck

I feel like the above wouldn't break CppCheck for anyone?

Many thanks, looking forward to any thoughts or feedback :)

[therisktaker]

Hi, just use --error-exitcode=1 and Cppcheck will exit with code = 1 when find any issue. That's enough to integrate in CI.

[Nathan]

Oh awesome. That could work. Are there similar for style, warning, etc?

[therisktaker]

I belive not, Cppcheck with exit with provided code if any issue will be found. So you need to clear your code or suppress warnings or use specific group of checks in --enable=x.

[Nathan]

Hmm. Thinking about --error-exitcode, I'd find this tool much more informative if it set ERRORLEVEL as PVS-Studio does. This allows PVS-Studio to exit 0 whilst still providing me with some information as to what went wrong:

      set /A FilesFail = "(%errorlevel% & 1) / 1"
      set /A GeneralExeption = "(%errorlevel% & 2) / 2"
      set /A IncorrectArguments = "(%errorlevel% & 4) / 4"
      set /A FileNotFound = "(%errorlevel% & 8) / 8"
      set /A IncorrectCfg = "(%errorlevel% & 16) / 16"
      set /A InvalidSolution = "(%errorlevel% & 32) / 32"
      set /A IncorrectExtension = "(%errorlevel% & 64) / 64"
      set /A IncorrectLicense = "(%errorlevel% & 128) / 128"
      set /A AnalysisDiff = "(%errorlevel% & 256) / 256"
      set /A SuppressFail = "(%errorlevel% & 512) / 512"
      set /A LicenseRenewal = "(%errorlevel% & 1024) / 1024"
      if %FilesFail% == 1 echo ##vso[task.logissue type=error]PVS-Studio Analysis: an error (crash) of the analyzer when checking one of the files.
      if %GeneralExeption% == 1 echo ##vso[task.logissue type=error]PVS-Studio Analysis: a general (non-specific) error during the analyzer operation, intercepted exception during operation.
      if %IncorrectArguments% == 1 echo ##vso[task.logissue type=error]PVS-Studio Analysis: some of the passed command-line arguments are incorrect.
      if %FileNotFound% == 1 echo ##vso[task.logissue type=error]PVS-Studio Analysis: the specified project, solution, or analyzer settings file wasn't found.
      if %IncorrectCfg% == 1 echo ##vso[task.logissue type=error]PVS-Studio Analysis: the specified configuration and / or platform weren't found in the solution file.
      if %InvalidSolution% == 1 echo ##vso[task.logissue type=error]PVS-Studio Analysis: the solution or project file isn't supported or contains errors.
      if %IncorrectExtension% == 1 echo ##vso[task.logissue type=error]PVS-Studio Analysis: invalid extension of the solution or project being checked.
      if %IncorrectLicense% == 1 echo ##vso[task.logissue type=error]PVS-Studio Analysis: invalid or expired analyzer license.
      if %AnalysisDiff% == 1 echo ##vso[task.logissue type=error]PVS-Studio Analysis: potential errors were found in the code being checked.
      if %SuppressFail% == 1 echo ##vso[task.logissue type=error]PVS-Studio Analysis: an error occurred while performing message suppression. 
      if %LicenseRenewal% == 1 echo ##vso[task.logissue type=error]PVS-Studio Analysis: indicates that the analyzer license will expire within a month.

Because CppCheck exits with a non-zero exit code for any error, it means that I then have to bodge/kludge some "keep running the pipeline anyways" to actually be able to process the report xml (if it even exits, the cppcheck exit code does not tell me whether it does) or other log file and upload the results to our CI system: I don't want the user to have to trawl logs when I can e.g. convert CppCheck XML with Sarif.Multitool to .sarif and use it with tooling that works with many tools, and I don't want to write 'keep running anyway' conditions for processing the cppcheck report - because cppcheck exits 1 for any error, if cppcheck crashes, I will have a cascading mess of failure following it in the pipeline. Does this make sense?

[therisktaker]

In Cppcheck --error-exitcode is like take it or leave it.

My approach is quite radical - I want to have all problems solved - that's why I do analysis on every build and CI breaks when some issue occurs - you have to spend time cleaning the codebase, but after that maintenance is simple. So I don't see any value with passing reports further.

You can combine --error-exitcode with processing XML. When exit code = 0 then everything is ok. When exit code != 0 then some defects were found, then process XML for metrics.

[Nathan]

I appreciate the radical approach, I promise I'm aiming for the same thing :D

(rant about 1000s of devs with centuries of legacy code across 100s of products and providing a consistent developer experience across probably billions of lines of code)

So, if I used --error-exitcode=1 against a baseline (--cppcheck-build-dir) and no new errors/warnings/etc were discovered, would cppcheck exit 0 or 1?

If it would exit 0 when nothing new is found compared to a baseline, this could work as a gate for a PR build and I may be able to live with the kludgy yaml to facilitate it. As we've been back and forth I've also been bashing away at my yaml, so, I think I can make this work..