Hello all,
Cpp check is a great tool, and i need to valid it.
Is anywhere the sources files package used for test it?
(i don't need the cppcheck source file, only "fake" sources file that allow to check the detection of cppcheck)
thanks to all.
nicolas
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
If you want to have a test suite with small test programs then you can extract the testcases from used in our testrunner. I do not have time to guide you much right now.. but you can look at the script cppcheck/tools/extracttests.py. I would suggest that you run it on cppcheck/test/testbufferoverrun.cpp for instance.
You can for instance use that to compare cppcheck against another tool. However please note the difference;
You can see that the other tool will not detect all bugs that Cppcheck finds.
You can NOT use it to see if Cppcheck detect all bugs that another tool finds.
You can not use it to see how many false positives Cppcheck will show because well it's test suite so of course Cppcheck handles that extremely well!
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
as I read it you want to have a test suite for static analysers.
I want to add that it's quite useless to create a synthetic test suite for static analysers. People have tried. I think the most famous test suites are Juliet and Toyota. They are not good. The faked bugs are unrealistic and does not test the code analysis properly.
In my humble opinion a more interesting test suite for static analysers would not contain any synthetic test cases at all but instead real code that has known bugs in it.
It's much more interesting to know if a tool finds real bugs than faked.
There are databases where you can see what the known vulnerabilities are in software. If I would create a test suite for code analysis I would look up known vulnerabilities in well known high quality open source software because then all the source code can be downloaded and the code is well written. Then I'd see if the static analysers could detect the known vulnerability in the source code.
For example these are known "overflow" security vulnerabilities in openssl:
tanks for your answer,
I agree with you about the fact that realistic test are done one real code, but for using tool we need to prove that tool is validated.
test runner package is exactly what i search for (even it is not perfect and of course will allways return true) it allow to see what is exactly test.
in my opinion, It's a big fail of software norm and standart, which prefer tat tool is tested even if the test are not plainty revelant than realistic test
Moreover cppCheck a tool ( even if it is a great tool!) and software validation shall still be done by dedicated tests and not only with static analysis.
thanks for all the job,
I will add some not detected error ("improve check") in trac as soon as i've get an account.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hello all,
Cpp check is a great tool, and i need to valid it.
Is anywhere the sources files package used for test it?
(i don't need the cppcheck source file, only "fake" sources file that allow to check the detection of cppcheck)
thanks to all.
nicolas
If you want to have a test suite with small test programs then you can extract the testcases from used in our testrunner. I do not have time to guide you much right now.. but you can look at the script cppcheck/tools/extracttests.py. I would suggest that you run it on cppcheck/test/testbufferoverrun.cpp for instance.
You can for instance use that to compare cppcheck against another tool. However please note the difference;
You can see that the other tool will not detect all bugs that Cppcheck finds.
You can NOT use it to see if Cppcheck detect all bugs that another tool finds.
You can not use it to see how many false positives Cppcheck will show because well it's test suite so of course Cppcheck handles that extremely well!
as I read it you want to have a test suite for static analysers.
I want to add that it's quite useless to create a synthetic test suite for static analysers. People have tried. I think the most famous test suites are Juliet and Toyota. They are not good. The faked bugs are unrealistic and does not test the code analysis properly.
In my humble opinion a more interesting test suite for static analysers would not contain any synthetic test cases at all but instead real code that has known bugs in it.
It's much more interesting to know if a tool finds real bugs than faked.
There are databases where you can see what the known vulnerabilities are in software. If I would create a test suite for code analysis I would look up known vulnerabilities in well known high quality open source software because then all the source code can be downloaded and the code is well written. Then I'd see if the static analysers could detect the known vulnerability in the source code.
For example these are known "overflow" security vulnerabilities in openssl:
https://www.cvedetails.com/vulnerability-list/vendor_id-217/product_id-383/opov-1/Openssl-Openssl.html
If anybody creates such a test suite with real code and runs Cppcheck and other tools on it.. I think it would be really cool!
Last edit: Daniel Marjamäki 2018-10-02
tanks for your answer,
I agree with you about the fact that realistic test are done one real code, but for using tool we need to prove that tool is validated.
test runner package is exactly what i search for (even it is not perfect and of course will allways return true) it allow to see what is exactly test.
in my opinion, It's a big fail of software norm and standart, which prefer tat tool is tested even if the test are not plainty revelant than realistic test
Moreover cppCheck a tool ( even if it is a great tool!) and software validation shall still be done by dedicated tests and not only with static analysis.
thanks for all the job,
I will add some not detected error ("improve check") in trac as soon as i've get an account.
Great! Looking forward to that.
To get a Trac account you create a htpasswd hash. You can create it online (google "htpasswd generator").
Then send that htpasswd hash to me.
Hi Nicolas,
Have you developed any test suite to validate cppcheck ?