I am experimenting with a webserver that will execute the test-my-pr script on PRs. I want more or less that it will provide daca@home feedback directly when somebody works on a PR.
It is only executed on open non-draft PRs that have a successful CI.
It will execute on 1000 packages if the PR modifies tokenizer or valueflow. Otherwise it will execute on 100 packages. When I started it , it would count packages even if they did not have C/C++ code that is fixed now..
AI-generated verdicts will be provided for a random subset of the warnings (up to 20 or so). Those should be taken with a grain of salt for now.
Please provide feedback. Do you think I should tweak the reports or output somehow?
Last edit: Daniel Marjamäki 2026-10-06
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
It's still being processed. If we will have trouble to keep up with open PRs I think we can launch more runner VMs.
It might be low with 100/1000 .. I will see if it makes sense to increase these values later. At least if a runner is idle it could fill up more packages..
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
One thing I recomend when it comes to AI is to keep internal metrics so that you can better help you evluete AI future solutions. That includes keeping track of the work done by AI which you can put a tag in an issue or a tag in the git commit message. You will want to be able to match that tag to a prompt, token usage and model version number. That way if you want to make changes later to your AI process you can test it aginst past solutions to see if the change adds value.
On a side note one thing I never got around to was trying to come up with better usage metrics for valueflow. While I was mainly looking for perfomance use case issues. I think there might be some value in value flow code coverage vs the debian packages. I know GCC 14 added a new feature called MCDC coverage -fcondition-coverage -fprofile-arcs -ftest-coverage It might make since to run this vs the debian packages maybe once a release because we might find a better core set of packages to run So instead of 100 and 1000 you could 70 and 850. You might find some long run time pages are just using the same paths in the value flow vs a lot of diffrent paths. I think as an excersize I can run cppcheck with the GCC MCDC vs serval debian packages and post my finding and output in a new thread if it provides anything useful.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I think one thing that matters a lot is how easy is it for you to integrate into your workflow. For me given that I learned to code inside a unix terminal and never evolved to using an IDE to code with AI isnt to difficult to record my interactions with. The script command https://www.redhat.com/en/blog/linux-script-command does alot of the heavy lifting. I also write a bit of data like starting and stopping git hash as well as the ai model version and token usage to a file. That way I can restore the state and test with a different model. In my mind I think this is useful to see which kind of AI workloads that can be shifted to slower local AI models and save tokens for fast interactive sessions when I am working directly with AI. For the most part I am only tracking my use of AI. I have given some thought to how I would do it from a software project level by using a git submodule. Lets take cppcheck as an example. I would have my AI meta data repo with cppcheck as a submodule to that repo. I even think some pre commit hooks could be used to help automate some data collection in the AI meta data repo. Also this helps with comparisons because cppcheck lives outside the repo. You need to do a 3 way diff to evaluate different AI models. To me I like all the AI meta data out side of the main project repo that way it doesn't pollute the main project repo. I could probably throw something together if you are intereste, but ever since vscode has come out as a popular IDE I see fewer people interested in terminal based solutions.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I am experimenting with a webserver that will execute the test-my-pr script on PRs. I want more or less that it will provide daca@home feedback directly when somebody works on a PR.
http://ec2-16-170-140-253.eu-north-1.compute.amazonaws.com/
It is only executed on open non-draft PRs that have a successful CI.
It will execute on 1000 packages if the PR modifies tokenizer or valueflow. Otherwise it will execute on 100 packages. When I started it , it would count packages even if they did not have C/C++ code that is fixed now..
AI-generated verdicts will be provided for a random subset of the warnings (up to 20 or so). Those should be taken with a grain of salt for now.
Please provide feedback. Do you think I should tweak the reports or output somehow?
Last edit: Daniel Marjamäki 2026-10-06
Having a link to the actual PR would be nice (instead of/additional to linking to the AWS files).
How is reanalysis supposed to work? E.g. for #8918, the latest commit shown is from Oct 5, while there is a newer commit from Oct 7 already (https://github.com/cppcheck-opensource/cppcheck/pull/8918/changes/97161058bc8dbc7b18acf69ae1cf324a21507f65).
Good idea 👍
Having the user name in a column would be nice.
Also, some PRs seem to be missing. Are they still being processed?
Last edit: CHR 2026-10-07
It's still being processed. If we will have trouble to keep up with open PRs I think we can launch more runner VMs.
It might be low with 100/1000 .. I will see if it makes sense to increase these values later. At least if a runner is idle it could fill up more packages..
One thing I recomend when it comes to AI is to keep internal metrics so that you can better help you evluete AI future solutions. That includes keeping track of the work done by AI which you can put a tag in an issue or a tag in the git commit message. You will want to be able to match that tag to a prompt, token usage and model version number. That way if you want to make changes later to your AI process you can test it aginst past solutions to see if the change adds value.
On a side note one thing I never got around to was trying to come up with better usage metrics for valueflow. While I was mainly looking for perfomance use case issues. I think there might be some value in value flow code coverage vs the debian packages. I know GCC 14 added a new feature called MCDC coverage -fcondition-coverage -fprofile-arcs -ftest-coverage It might make since to run this vs the debian packages maybe once a release because we might find a better core set of packages to run So instead of 100 and 1000 you could 70 and 850. You might find some long run time pages are just using the same paths in the value flow vs a lot of diffrent paths. I think as an excersize I can run cppcheck with the GCC MCDC vs serval debian packages and post my finding and output in a new thread if it provides anything useful.
how do you suggest that I do that?
I think one thing that matters a lot is how easy is it for you to integrate into your workflow. For me given that I learned to code inside a unix terminal and never evolved to using an IDE to code with AI isnt to difficult to record my interactions with. The script command https://www.redhat.com/en/blog/linux-script-command does alot of the heavy lifting. I also write a bit of data like starting and stopping git hash as well as the ai model version and token usage to a file. That way I can restore the state and test with a different model. In my mind I think this is useful to see which kind of AI workloads that can be shifted to slower local AI models and save tokens for fast interactive sessions when I am working directly with AI. For the most part I am only tracking my use of AI. I have given some thought to how I would do it from a software project level by using a git submodule. Lets take cppcheck as an example. I would have my AI meta data repo with cppcheck as a submodule to that repo. I even think some pre commit hooks could be used to help automate some data collection in the AI meta data repo. Also this helps with comparisons because cppcheck lives outside the repo. You need to do a 3 way diff to evaluate different AI models. To me I like all the AI meta data out side of the main project repo that way it doesn't pollute the main project repo. I could probably throw something together if you are intereste, but ever since vscode has come out as a popular IDE I see fewer people interested in terminal based solutions.
Depending on how many tokens you want to use here are other things you can ask AI about a PR.