CppCheck validation package

[Delplanque nicolas]

Hello all,
Cpp check is a great tool, and i need to valid it.
Is anywhere the sources files package used for test it?
(i don't need the cppcheck source file, only "fake" sources file that allow to check the detection of cppcheck)

thanks to all.
nicolas

[Daniel Marjamäki]

If you want to have a test suite with small test programs then you can extract the testcases from used in our testrunner. I do not have time to guide you much right now.. but you can look at the script cppcheck/tools/extracttests.py. I would suggest that you run it on cppcheck/test/testbufferoverrun.cpp for instance.

You can for instance use that to compare cppcheck against another tool. However please note the difference;
You can see that the other tool will not detect all bugs that Cppcheck finds.
You can NOT use it to see if Cppcheck detect all bugs that another tool finds.

You can not use it to see how many false positives Cppcheck will show because well it's test suite so of course Cppcheck handles that extremely well!

[Daniel Marjamäki]

as I read it you want to have a test suite for static analysers.

I want to add that it's quite useless to create a synthetic test suite for static analysers. People have tried. I think the most famous test suites are Juliet and Toyota. They are not good. The faked bugs are unrealistic and does not test the code analysis properly.

In my humble opinion a more interesting test suite for static analysers would not contain any synthetic test cases at all but instead real code that has known bugs in it.

It's much more interesting to know if a tool finds real bugs than faked.

There are databases where you can see what the known vulnerabilities are in software. If I would create a test suite for code analysis I would look up known vulnerabilities in well known high quality open source software because then all the source code can be downloaded and the code is well written. Then I'd see if the static analysers could detect the known vulnerability in the source code.

For example these are known "overflow" security vulnerabilities in openssl:

https://www.cvedetails.com/vulnerability-list/vendor_id-217/product_id-383/opov-1/Openssl-Openssl.html

If anybody creates such a test suite with real code and runs Cppcheck and other tools on it.. I think it would be really cool!

[Delplanque nicolas]

tanks for your answer,
I agree with you about the fact that realistic test are done one real code, but for using tool we need to prove that tool is validated.
test runner package is exactly what i search for (even it is not perfect and of course will allways return true) it allow to see what is exactly test.
in my opinion, It's a big fail of software norm and standart, which prefer tat tool is tested even if the test are not plainty revelant than realistic test

Moreover cppCheck a tool ( even if it is a great tool!) and software validation shall still be done by dedicated tests and not only with static analysis.

thanks for all the job,
I will add some not detected error ("improve check") in trac as soon as i've get an account.

[Daniel Marjamäki]

I will add some not detected error ("improve check") in trac as soon as i've get an account.

Great! Looking forward to that.

To get a Trac account you create a htpasswd hash. You can create it online (google "htpasswd generator").

Then send that htpasswd hash to me.

[kpk]

Hi Nicolas,
Have you developed any test suite to validate cppcheck ?