do you see in threadexecutor why suppressions become different? I assume you run the same cppcheck binary twice and get different suppressions? If you can point out the problem.. let me know.
we have separate implementations for linux and windows. in linux, processes are used. in windows, threads are used.
Nowadays .. we could use std::thread so we get the same code for windows and linux. pkeus has even implemented this so we can do that if we want. However unfortunately I believe we loose the handling of crashes in linux if we do that.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
My first suspicion was that there is a difference between the Linux and Windows code, but I have not confirmed this. I saw that pipes are being used in FORK, but could not identify the equivalent mechanism in WIN. I'll have to debug this some more.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
when threads are used the errors can be passed directly because there is shared memory. When processes are used we need to send the information between processes using such mechanism.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
The problem seems to be related to the way suppressions accumulate in CppCheck::mSettings.nomsg.
I have instrumented Preprocessor::inlineSuppressions() to show where the suppressions come from. They can be (1) stored already, (2) parsed from the input token list (tokens1 from checkFile() or (3) parsed from Preprocessor::mTokenLists.
When I run cppcheck on a single file, I get e.g. 0 0 13 (13 suppressions from mTokenLists. When the identical file gets analyzed as part of the whole directory, I might get 16 16 16 (16 suppressions already stored, none parsed). The additional suppressions seem to stem from files that the respective thread has already analysed.
Comparing ThreadExecutor::check()(FORK) and threadProc() (WIN), the former forks for every file and then creates a new CppCheckobject, whereas one object per thread lives for the whole duration of the analysis in the latter function. That explains why the problem only exists on Windows. The fix would be to move the fileChecker object inside the for (;;) loop. Does this make sense?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I can confirm that this fixes the issue for me. All existing .a1 are getting picked up, and no re-analysis takes place in subsequent runs, resulting in <1 min of run time.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
It sounds reasonable to move the fileCheck object. That will match the fork behavior better. Can somebody also check if this works in the GUI also.
It would be nice with some test that ensures that the output is valid xml. And that rerunning will produce identical hash (the whole file should be identical but well if we ever want to add some timestamp or something then it would be better to check the hash).
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Fixed in https://github.com/danmar/cppcheck/pull/3405. I think the existence of this bug illustrates why a single implementation for all platforms would be preferable, especially if someone has done the work already.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Yes. But then we have to sacrifice the crash handling and that is pretty nice. Especially getting a backtrace in daca@home.
Maybe replace the thread implementation to start with, the windows-specific code can be replaced with generic code. Then we can choose in linux if we want to use threads or processes. It will be a step in the right direction.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi,
I read the discussion and I checked in the manual and in "news" but I cannot find any tips regarding the "--cppcheck-build-dir" option. I tried it with 2.3 and I got an error. Is it something under development?
thanks
michele
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I was just adding the --cppcheck-build-dir without the last part. Very interesting, I tested it and it works fine, thanks!
On the manul I didn't find anything about it, am I wrong?
thanks
michele
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I promised to come back with timings. Note that I am using the unpatched v2.5. The time to completion can go down to 40minutes with the --cppcheck-build-dir option. Its clear from looking at the spawned processes with 'ps' that there is one child process taking much longer than the others (I guess some particular file is difficult to digest)
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I would really appreciate some feedback on this.
do you see in threadexecutor why suppressions become different? I assume you run the same cppcheck binary twice and get different suppressions? If you can point out the problem.. let me know.
we have separate implementations for linux and windows. in linux, processes are used. in windows, threads are used.
Nowadays .. we could use std::thread so we get the same code for windows and linux. pkeus has even implemented this so we can do that if we want. However unfortunately I believe we loose the handling of crashes in linux if we do that.
My first suspicion was that there is a difference between the Linux and Windows code, but I have not confirmed this. I saw that pipes are being used in FORK, but could not identify the equivalent mechanism in WIN. I'll have to debug this some more.
when threads are used the errors can be passed directly because there is shared memory. When processes are used we need to send the information between processes using such mechanism.
Makes sense, thanks.
that sounds like a bug. I can't see a good reason for suppressions to change.
The problem seems to be related to the way suppressions accumulate in
CppCheck::mSettings.nomsg.I have instrumented
Preprocessor::inlineSuppressions()to show where the suppressions come from. They can be (1) stored already, (2) parsed from the input token list (tokens1fromcheckFile()or (3) parsed fromPreprocessor::mTokenLists.When I run cppcheck on a single file, I get e.g.
0 0 13(13 suppressions frommTokenLists. When the identical file gets analyzed as part of the whole directory, I might get16 16 16(16 suppressions already stored, none parsed). The additional suppressions seem to stem from files that the respective thread has already analysed.Comparing
ThreadExecutor::check()(FORK) andthreadProc()(WIN), the former forks for every file and then creates a newCppCheckobject, whereas one object per thread lives for the whole duration of the analysis in the latter function. That explains why the problem only exists on Windows. The fix would be to move thefileCheckerobject inside thefor (;;)loop. Does this make sense?I can confirm that this fixes the issue for me. All existing
.a1are getting picked up, and no re-analysis takes place in subsequent runs, resulting in <1 min of run time.excellent work!
It sounds reasonable to move the
fileCheckobject. That will match the fork behavior better. Can somebody also check if this works in the GUI also.It would be nice with some test that ensures that the output is valid xml. And that rerunning will produce identical hash (the whole file should be identical but well if we ever want to add some timestamp or something then it would be better to check the hash).
Fixed in https://github.com/danmar/cppcheck/pull/3405. I think the existence of this bug illustrates why a single implementation for all platforms would be preferable, especially if someone has done the work already.
Yes. But then we have to sacrifice the crash handling and that is pretty nice. Especially getting a backtrace in daca@home.
Maybe replace the thread implementation to start with, the windows-specific code can be replaced with generic code. Then we can choose in linux if we want to use threads or processes. It will be a step in the right direction.
That should be relatively easy, see here: https://github.com/danmar/cppcheck/pull/3408
Hi,
I read the discussion and I checked in the manual and in "news" but I cannot find any tips regarding the "--cppcheck-build-dir" option. I tried it with 2.3 and I got an error. Is it something under development?
thanks
michele
It has been available for years. Should be stable in 2.3 also. What error did you get? What command line did you use? Example use:
If you do not remove/change anything in
bthen the next time you execute cppcheck you should get a significant speedup:My next scans are due 4 August and 6 August; I'll report the timings after.
I was just adding the --cppcheck-build-dir without the last part. Very interesting, I tested it and it works fine, thanks!
On the manul I didn't find anything about it, am I wrong?
thanks
michele
I am looking at the manual now. You are right. I will fix it, there will be a chapter about the cppcheck build dir asap.
Last edit: Daniel Marjamäki 2021-08-02
Thank you very much. Great update I missed.
Michele
I promised to come back with timings. Note that I am using the unpatched v2.5. The time to completion can go down to 40minutes with the --cppcheck-build-dir option. Its clear from looking at the spawned processes with 'ps' that there is one child process taking much longer than the others (I guess some particular file is difficult to digest)
Maybe you can also test with a nightly build. Performance has definitely improved in the meantime.