Add samples/id/good|bad.cpp

[aggro80]

Add samples/id/good|bad.cpp for each error cppcheck detects.

e.g.
samples/uninitdata/good.cpp
samples/uninitdata/bad.cpp

Content of these files is valid C or C++ code. bad.cpp contains source which will provide error/warning in cppcheck. good.cpp contains the fixed version of the same code that gives no warnings.

Advantages:
- It can be used to test cppcheck and competitors
the error code if user requests
- We can probably automate a test to detect if some test is missing those files or if some files exist for test which has been removed, making maintenance easier.
- It can be the raw data to create
-- Online list of errors we detect (with sample codes)
-- Improve error messages e.g. in gui by providing sample code with
-- Anything else we might need in the future. E.g. with a small script we can probably generate a class from the files to provide the data to any program.

Note the current (but very small) list of ErrorMessages and sample codes.

[seb777]

There are already unit tests in cppcheck. I think that's enough.

Try to compile it with the command "make test".

[danielmarjamaki]

There are already unit tests in cppcheck. I think that's enough.

I certainly agree that we will continue to use unit tests for regression tests.

But when Reijo said "It can be used to test cppcheck and competitors", I think he talked about such cases:

• If a user wants to test Cppcheck he prefer to play with a real code sample instead of our unit tests.
• If I want to compare Cppcheck against for instance pc-lint, I must have some sample code.

[aggro80]

The "comparing cppcheck to competitors" part is just optional benefit, which allows us to test weather competitors give false positives and how many of the errors we detect, they fail to detect. This should IMHO be done just because it comes with very little work when we complete the actual goal.

The actual goal for this is to create documentation for the user. We get quite often questions at least in irc where users ask "what does this error mean, how do I fix it". This task is about creating the raw data, which we can then manipulate to create the documentation in proper format and show it to users in proper places.

[seb777]

There are already test suites. See the following link: http://samate.nist.gov/SRD/testsuite.php

If you want to make a comparative study of various code analysis tools, feel free to post your comments and results. This may be interested.

But, I think that these test suites are not really beneficial for educational use. The examples are not quite enough real cases. They simply show the relevance of the tools in the error detection code.

[edward-san]

Who the hell wants to download almost 85 (the zip file, 450 megs unzipped) megs of test cases?

[danielmarjamaki]

There are already test suites.

It seems to me that you have misunderstood twice what this ticket is about. It is not about regression testing and it is not about comparing cppcheck against other tools.

Imagine that a user wants to see sample code that trigger Cppcheck to show "variableScope" warnings. The samate test suite can't be used for this because:

• it is grouped according to CWE instead of Cppcheck-ID so it would not be trivial to find corresponding samples.
• it probably doesn't have specific samples for every Cppcheck-ID.

Our "make test" can't be used at all because in most cases a user only has installed the cppcheck binary. And for those who build from source code it's a lot more difficult to write a unit test than using the cppcheck binary.

[seb777]

OK the goal is to write an expanded documentation of numerous examples (good and bad code) on the various checks. It sound a good idea in this way.

[danielmarjamaki]

ok

[aggro80]

The work has now started:
https://github.com/danmar/cppcheck/commit/3066bb3430c8ce7dab1dda27f04081d2492ecb5a

[danielmarjamaki]

ok good.

What are the requirements for a sample?

• It must be compilable by a standard C/C++ compiler, right?
• how well commented should it be?
• should it be as short as possible or should it have some functionality?

[aggro80]

- Yes, I think we should aim for compilable samples. If for nothing other purpose, at least the samples should always work, even if we add some additional checks to cppcheck to prevent it from checking incomplete code.
- Comments and functionality are completely optional IMHO. But if you ever find a programmer to whom the code samples are not obvious with the error message, then we probably need to improve either the messages or the samples.

But this is just my opinion, so feel free to disagree ;)