|
From: Nicholas N. <nj...@cs...> - 2007-10-05 00:50:07
|
Hi, Julian and I have been discussing for a while the idea of creating a category of "experimental" tools in the Valgrind source tree. These would be tools that are not necessarily mature, but may be of interest to people. This would make it much easier for users to try out experimental tools -- they would be included in the Valgrind distribution -- which in turn might accelerate their development. The core Valgrind developers would maintain experimental tools at the minimum level of keeping them compiling, eg. if the core/tool interface changes, as it does occasionally. But each tool would have to have a nominated maintainer who would be reponsible for responding to bug reports, feature requests, and who would continue development. These tools would be given an "exp-" prefix to their name to indicate their experimental nature, and they would be marked clearly as experimental on the website and in the manual. Experimental tools could lose the "exp-" prefix once they reached a certain level of maturity and popularity. The tools would have to be licensed as GPLv2-or-later, like the rest of Valgrind. We are interested in hearing people's opinions on this idea. We are also interested to hear if anyone has a tool they would like to be considered for inclusion as an experimental tool. Nick |
|
From: Bart V. A. <bar...@gm...> - 2007-10-05 05:29:18
|
On 10/5/07, Nicholas Nethercote <nj...@cs...> wrote: > Julian and I have been discussing for a while the idea of creating a > category of "experimental" tools in the Valgrind source tree. These > would be tools that are not necessarily mature, but may be of interest > to people. This would make it much easier for users to try out > experimental tools -- they would be included in the Valgrind distribution > -- > which in turn might accelerate their development. > I'm in favor of this proposal, and would appreciate it if drd could be added as an experimental tool. Since the drd and thrcheck tools use different algorithms for detecting data races, it can be interesting to compare the output of the two tools and their performance. Bart. |
|
From: Robert W. <rj...@du...> - 2007-10-05 17:08:18
|
Might this be a suitable location for omega, too? Assuming someone is OK with maintaining it enough to keep it compiling at least? On Oct 4, 2007, at 10:29 PM, Bart Van Assche wrote: > On 10/5/07, Nicholas Nethercote <nj...@cs...> wrote: > > Julian and I have been discussing for a while the idea of creating a > category of "experimental" tools in the Valgrind source tree. These > would be tools that are not necessarily mature, but may be of interest > to people. This would make it much easier for users to try out > experimental tools -- they would be included in the Valgrind > distribution -- > which in turn might accelerate their development. > > I'm in favor of this proposal, and would appreciate it if drd could > be added as an experimental tool. Since the drd and thrcheck tools > use different algorithms for detecting data races, it can be > interesting to compare the output of the two tools and their > performance. > > Bart. > ---------------------------------------------------------------------- > --- > This SF.net email is sponsored by: Splunk Inc. > Still grepping through log files to find problems? Stop. > Now Search log events and configuration files using AJAX and a > browser. > Download your FREE copy of Splunk now >> http://get.splunk.com/ > _______________________________________________ > Valgrind-developers mailing list > Val...@li... > https://lists.sourceforge.net/lists/listinfo/valgrind-developers |
>>>>> "NN" == Nicholas Nethercote <nj...@cs...> writes:
NN> Hi,
NN> Julian and I have been discussing for a while the idea of creating
NN> a category of "experimental" tools in the Valgrind source tree.
NN> These would be tools that are not necessarily mature, but may be
NN> of interest to people. This would make it much easier for users
NN> to try out experimental tools -- they would be included in the
NN> Valgrind distribution -- which in turn might accelerate their
NN> development.
[...]
NN> We are interested in hearing people's opinions on this idea. We
NN> are also interested to hear if anyone has a tool they would like
NN> to be considered for inclusion as an experimental tool.
I think this sounds like a great idea. Having more tools in the source
tree will make it easier on both sides to keep the core and the tools
working together, and having more tools in the distribution will
advertise Valgrind's flexibility as a framework and potentially, as
you suggest, feed a virtuous cycle of tool use and development.
More broadly, though, I think that similar reasons suggest that other
forms of development on Valgrind could benefit from being closer in
sync with the core, even if it doesn't make sense for them to appear
in the distribution. This includes incomplete architecture and OS
ports, experimental core features, tools that need core changes, and
tools that have only specialized uses. Would it be reasonable to
invite more things like this to live on branches in the SVN
repository?
I think of the Fjalar/Kvasir code I maintain as falling into the
latter category: it only useful to Daikon users, or potentially to
authors of similar tools, not to general developer-users. And it's
certainly way too easy for it to fall out of sync with the core living
as it does in a local CVS repository. Some of that is general laziness
that won't be helped by any organizational change (merging updates
from the core is a non-automatable task with only delayed benefits),
but I think fostering a culture of more in-sync development would be
good for everyone.
-- Stephen
|
|
From: Nicholas N. <nj...@cs...> - 2007-10-08 00:20:12
|
On Fri, 5 Oct 2007, Stephen McCamant wrote: > More broadly, though, I think that similar reasons suggest that other > forms of development on Valgrind could benefit from being closer in > sync with the core, even if it doesn't make sense for them to appear > in the distribution. This includes incomplete architecture and OS > ports, experimental core features, tools that need core changes, and > tools that have only specialized uses. Would it be reasonable to > invite more things like this to live on branches in the SVN > repository? > > I think of the Fjalar/Kvasir code I maintain as falling into the > latter category: it only useful to Daikon users, or potentially to > authors of similar tools, not to general developer-users. And it's > certainly way too easy for it to fall out of sync with the core living > as it does in a local CVS repository. Some of that is general laziness > that won't be helped by any organizational change (merging updates > from the core is a non-automatable task with only delayed benefits), > but I think fostering a culture of more in-sync development would be > good for everyone. I think this idea definitely has merit, and it certainly would make life easier for people like you working on related things. The main thing that concerns me is that if something goes on a branch, is there an implicit idea that it could/would one day go into the core? For example, at http://www.valgrind.org/info/platforms.html we make it clear that we don't want to support a wide range of platforms, because the cost:benefit ratio is not favourable. But some people are working on ports to some of the platforms that we're less interested in (eg. the BSDs). If we made a branch for such ports it might make it harder to maintain this clear position. It's a social/political issue rather than a technical one, perhaps if we had very clear rules it could be workable. Ultimately Julian has the final say, perhaps he has some more to add. Nick |
|
From: Dirk M. <dm...@gm...> - 2007-10-07 02:11:40
|
On Friday, 5. October 2007, Nicholas Nethercote wrote: > Julian and I have been discussing for a while the idea of creating a > category of "experimental" tools in the Valgrind source tree. These > would be tools that are not necessarily mature, but may be of interest > to people. This would make it much easier for users to try out > experimental tools -- they would be included in the Valgrind distribution > -- which in turn might accelerate their development. some of the experimental tools available require significant (and conflicting) changes to the core, which is the reason e.g. distributions have troubles packaging the addon experimental tools. > These tools would be given an "exp-" prefix to their name to indicate > their experimental nature, and they would be marked clearly as experimental > on the website and in the manual. Experimental tools could lose the "exp-" > prefix once they reached a certain level of maturity and popularity. what about documentation that is written with the exp- prefix, which then becomes out of date? broken user habits/custom scripts? Tagging valgrind assertions/crashdumps/warning output with a clear indication that an experimental plugin was run (sort of tainting-concept) seems more useful to me. Greetings, Dirk |
|
From: Nicholas N. <nj...@cs...> - 2007-10-08 00:01:16
|
On Fri, 5 Oct 2007, Robert Walsh wrote: >> I'm in favor of this proposal, and would appreciate it if drd could be >> added as an experimental tool. Since the drd and thrcheck tools use >> different algorithms for detecting data races, it can be interesting to >> compare the output of the two tools and their performance. Yes, DRD is an obvious candidate for inclusion. Can you remind me -- does it work with the current core? If not, how big are the changes you'd need? > Might this be a suitable location for omega, too? Assuming someone is OK > with maintaining it enough to keep it compiling at least? If someone volunteered to be a maintainer, it would be suitable, IMO. Without a maintainer, I'm less keen because simple keep-it-compiling-only maintenance doesn't work that well in practice, as we saw with Helgrind. Nick |
|
From: Julian S. <js...@ac...> - 2007-10-08 08:42:52
|
> > Might this be a suitable location for omega, too? Assuming someone is OK > > with maintaining it enough to keep it compiling at least? > > If someone volunteered to be a maintainer, it would be suitable, IMO. > Without a maintainer, I'm less keen because simple keep-it-compiling-only > maintenance doesn't work that well in practice, as we saw with Helgrind. I think Omega would be a good candidate for an exp- tool. We do still need a maintainer though, else it will inevitably become unusable as the core changes. Which is a shame as there has been a constant trickle of requests about it over the past few months. With Helgrind we were kinda cheating w.r.t. keeping it compiling. It only compiles because various bits of code (eg, the main instrumentation function) had been #if 0'd or completely removed. And so it doesn't even really compile. J |
|
From: Nicholas N. <nj...@cs...> - 2007-10-08 00:12:18
|
On Sun, 7 Oct 2007, Dirk Mueller wrote: >> Julian and I have been discussing for a while the idea of creating a >> category of "experimental" tools in the Valgrind source tree. These >> would be tools that are not necessarily mature, but may be of interest >> to people. This would make it much easier for users to try out >> experimental tools -- they would be included in the Valgrind distribution >> -- which in turn might accelerate their development. > > some of the experimental tools available require significant (and conflicting) > changes to the core, which is the reason e.g. distributions have troubles > packaging the addon experimental tools. Is that an argument for or against the proposal, or just a general comment? Tools that required big core changes would be less likely to be accepted, although it would be decided on a case-by-case basis. GCC provides a reasonable comparison point -- they have a lot of different platforms, some of which are "primary" and some are "secondary", and (AIUI) they're generally happy to add secondary platforms so long as someone maintains them, but new platforms generally have to fit into the existing framework. >> These tools would be given an "exp-" prefix to their name to indicate >> their experimental nature, and they would be marked clearly as experimental >> on the website and in the manual. Experimental tools could lose the "exp-" >> prefix once they reached a certain level of maturity and popularity. > > what about documentation that is written with the exp- prefix, which then > becomes out of date? broken user habits/custom scripts? That doesn't worry me particularly. We've broken backward compatibility in the past on such things. Also, the possibility of a future name change would be made clear in the documentation on experimental tools, so there'd be an inherent "user beware" aspect to them. I also don't think most of them would be particularly widely used -- Memcheck accounts for over 80% of Valgrind usage. > Tagging valgrind assertions/crashdumps/warning output with a clear indication > that an experimental plugin was run (sort of tainting-concept) seems more > useful to me. We could do that in addition. Some extra information that might be of interest: at one point Julian and I were considering having two versions of the distribution, "valgrind" and "valgrind+extras", due to concerns that if the experimental tools were of a lower quality that they might hurt Valgrind's reputation. But that would make building/testing/installing more complicated, and it would reduce the exposure of the experimental tools, since lots of people would probably not install the extras. The "exp-" prefix seems like a good compromise because it means the experimental tools are widely available, but still making it very clear that they are experimental. Nick |
|
From: Bart V. A. <bar...@gm...> - 2007-10-08 06:58:10
|
On 10/8/07, Nicholas Nethercote <nj...@cs...> wrote: > >> I'm in favor of this proposal, and would appreciate it if drd could be > >> added as an experimental tool. Since the drd and thrcheck tools use > >> different algorithms for detecting data races, it can be interesting to > >> compare the output of the two tools and their performance. > > Yes, DRD is an obvious candidate for inclusion. Can you remind me -- does > it work with the current core? If not, how big are the changes you'd need? The changes needed to get the latest published drd patch working with the current core on the trunk are really minor -- one or two core functions called by drd were renamed in the core. However, both drd and thrcheck change the core and these changes overlap. Will the experimental tools be included in Valgrind before or after thrcheck is merged to the trunk ? Regards, Bart. |
|
From: Julian S. <js...@ac...> - 2007-10-08 08:35:14
|
> However, both drd and thrcheck change the core and these changes > overlap. Will the experimental tools be included in Valgrind before or > after thrcheck is merged to the trunk ? Probably around the same time. I expect to add to the trunk, core changes needed to support thrcheck, in the next two weeks or so. In which case that should support drd too. J |
|
From: Julian S. <js...@ac...> - 2007-10-08 08:32:07
|
> > some of the experimental tools available require significant (and > > conflicting) changes to the core, which is the reason e.g. distributions > > have troubles packaging the addon experimental tools. > > Is that an argument for or against the proposal, or just a general comment? > > Tools that required big core changes would be less likely to be accepted, > although it would be decided on a case-by-case basis. Yes, I'd want to be relatively conservative about changing the core. But my impression is that the core/tool interface is pretty effective at insulating tools from the core, and most tools will not need much by way of core changes. For those that do, we could put development on a branch so as to see what core changes will be necessary, then merge to trunk if they seem ok. > > Tagging valgrind assertions/crashdumps/warning output with a clear > > indication that an experimental plugin was run (sort of tainting-concept) > > seems more useful to me. > > We could do that in addition. In addition, yes. But not instead of the exp- name. For the most part tools which are under development don't crash; they just tend to produce results which are sometimes unreliable, inaccurate, etc. And so you want to make clear to users which tools are considered production-ready (you can rely on the output) and those which aren't (you have to be more careful in interpreting the output). J |
|
From: Dirk M. <dm...@gm...> - 2007-10-08 08:59:34
|
On Monday 08 October 2007, Julian Seward wrote: > And so you want > to make clear to users which tools are considered production-ready (you > can rely on the output) and those which aren't (you have to be more careful > in interpreting the output). Ok, in that case a --yes-I-accept-the-disclaimer option that you have to pass via e.g. environment or interactive question during startup would make it more clear to the user that the user should not put too much trust into the report. For me, a mere "exp-" prefix is ambiguous - it can mean that the tool might be unstable, not that it might be incorrect. Dirk |
|
From: Dirk M. <dm...@gm...> - 2007-10-08 08:47:29
|
On Monday 08 October 2007, Nicholas Nethercote wrote: > > some of the experimental tools available require significant (and > > conflicting) changes to the core, which is the reason e.g. distributions > > have troubles packaging the addon experimental tools. > Is that an argument for or against the proposal, or just a general comment? General comment. Although anything that makes this situation better would be highly welcome. > Tools that required big core changes would be less likely to be accepted, > although it would be decided on a case-by-case basis. how do you think should this look like? accept core changes in an #ifdef? accept core changes as long as they can not affect any of the non-experimental tools? Delay addition of an experimental plugin until it doesn't require core changes? > > what about documentation that is written with the exp- prefix, which then > > becomes out of date? broken user habits/custom scripts? > That doesn't worry me particularly. We've broken backward compatibility in > the past on such things. Also, the possibility of a future name change > would be made clear in the documentation on experimental tools, so there'd > be an inherent "user beware" aspect to them. Did you observe at all that someone questions valgrind's (memcheck/core) reputation because they found a bad plugin somewhere? If not so, why treat those experimental plugins as a second class citizen? > I also don't think most of > them would be particularly widely used -- Memcheck accounts for over 80% of > Valgrind usage. Thats true, but I thought one of the aspects of this proposal was to change that :) > Some extra information that might be of interest: at one point Julian and > I were considering having two versions of the distribution, "valgrind" and > "valgrind+extras", due to concerns that if the experimental tools were of a > lower quality that they might hurt Valgrind's reputation. Ok. If the other option is to do two branches (stable and experimental) in addition to the already stable and development code branches, then I'm all against avoiding yet another level of indirection/diversity. Greetings, Dirk |
|
From: Nicholas N. <nj...@cs...> - 2007-10-10 00:16:47
|
On Mon, 8 Oct 2007, Dirk Mueller wrote: >> Tools that required big core changes would be less likely to be accepted, >> although it would be decided on a case-by-case basis. > > how do you think should this look like? accept core changes in an #ifdef? > accept core changes as long as they can not affect any of the > non-experimental tools? Delay addition of an experimental plugin until it > doesn't require core changes? #ifdefs are a bad idea, IMO. The other two guidelines are possibilities. There won't be a hard-and-fast rule. For example, I think Bart wants/needs some minor changes for DRD. These seem pretty reasonable, and they would improve DRD's messages, and they're not likely to screw anything else up, so they're fine. Stephen's Fjalar changes (mostly to do with better debug info reading, IIUC) are much larger, and so less likely to be accepted, although if they are generally useful they could be. They would certainly require more scrutiny. Perhaps they could live in a branch for a while. In general, small changes are more likely to be acceptable than large changes, and general features are more likely to be acceptable than features that are more tool-specific. >>> what about documentation that is written with the exp- prefix, which then >>> becomes out of date? broken user habits/custom scripts? >> That doesn't worry me particularly. We've broken backward compatibility in >> the past on such things. Also, the possibility of a future name change >> would be made clear in the documentation on experimental tools, so there'd >> be an inherent "user beware" aspect to them. > > Did you observe at all that someone questions valgrind's (memcheck/core) > reputation because they found a bad plugin somewhere? No, but it's a possibility, and the "exp-" prefix is an attempt to avoid it. > If not so, why treat those experimental plugins as a second class citizen? Because they are second class citizens -- they haven't been widely used. Once they have been, if they seem good enough, the can be promoted to non-exp status. Dirk, just to clarify -- you've asked lots of questions about the details, which is good. But do you think it's a good idea in general? We have approvals from Bart and Stephen, and an implicit approval from Robert. Nick |
|
From: Josef W. <Jos...@gm...> - 2007-10-10 15:12:01
|
On Wednesday 10 October 2007, Nicholas Nethercote wrote: > On Mon, 8 Oct 2007, Dirk Mueller wrote: > > >> Tools that required big core changes would be less likely to be accepted, > >> although it would be decided on a case-by-case basis. > > > > how do you think should this look like? accept core changes in an #ifdef? > > accept core changes as long as they can not affect any of the > > non-experimental tools? Delay addition of an experimental plugin until it > > doesn't require core changes? > > #ifdefs are a bad idea, IMO. The other two guidelines are possibilities. Yes. FWIW, I am fine with branches. IMHO it is good to have a central place where experimental tools (and according core changes) are available. This way, the chance is higher that a new developer will improve on experimental tools to reach further stability. As tools are linked statically, it should be even possible to install tools using different versions of VG core next to each other; aside from incompatibilities in the launcher, which we would not allow. For experimental tools, IMHO a "exp-" is too cryptic. If you want to go this way, it should be "experimental-...". But instead, I just would abort with a good message when not run with an option "--yes-i-know-this-tool-is-experimental". Being interactive is bad when you want to run the tool from scripts. > Dirk, just to clarify -- you've asked lots of questions about the details, > which is good. But do you think it's a good idea in general? We have > approvals from Bart and Stephen, and an implicit approval from Robert. IMHO providing experimental tools is a good idea. Josef |
|
From: Dirk M. <dm...@gm...> - 2007-10-10 00:25:43
|
On Wednesday 10 October 2007, Nicholas Nethercote wrote: > Dirk, just to clarify -- you've asked lots of questions about the details, > which is good. But do you think it's a good idea in general? Yes of course, although except for the "exp" prefix, as pointed out already (experimental could mean a lot of different things, and the clear message "do not trust this plugins output" is imho not expressed by it). but as you said - the details need to be worked out. I just tried to discuss them to see that it really is a good idea in general :) Thanks, Dirk |
|
From: Robert W. <rj...@du...> - 2007-10-10 01:00:24
|
I'm OK with the idea - nothing implicit about that :-)
Not that I'm happy to whip this dead horse again, but one thing
that's kind of annoying is that a lot of this is working around
subversion. Subversion doesn't do branch updating well, so the
option to "create a branch for experimental tools that require core
changes" involves a lot more maintenance work than with a properly
distributed revision control system. Changesets have to be manually
merged and something outside the system needs to remember what was
merged so consistency can be maintained.
Here's a thought: if we're stuck with subversion, how about
maintaining core changes (and the experimental tools, too) as patches
managed by a patch management system like quilt?
http://savannah.nongnu.org/projects/quilt
Regards,
Robert.
|
|
From: Bart V. A. <bar...@gm...> - 2007-11-17 12:51:39
|
On Oct 5, 2007 1:50 AM, Nicholas Nethercote <nj...@cs...> wrote: > > Julian and I have been discussing for a while the idea of creating a > category of "experimental" tools in the Valgrind source tree. These > would be tools that are not necessarily mature, but may be of interest > to people. This would make it much easier for users to try out > experimental tools -- they would be included in the Valgrind distribution -- > which in turn might accelerate their development. > > The core Valgrind developers would maintain experimental tools at the > minimum level of keeping them compiling, eg. if the core/tool > interface changes, as it does occasionally. But each tool would have > to have a nominated maintainer who would be reponsible for responding to bug > reports, feature requests, and who would continue development. > > These tools would be given an "exp-" prefix to their name to indicate > their experimental nature, and they would be marked clearly as experimental > on the website and in the manual. Experimental tools could lose the "exp-" > prefix once they reached a certain level of maturity and popularity. > The tools would have to be licensed as GPLv2-or-later, like the rest of > Valgrind. I propose to use the prefix exp_ instead of exp-. My motivation is as follows: * The rule all-local from the top-level Makefile.am (which installs tool executable and redirections into the .in_place directory) fails for tools which have a hyphen in their name. * The core fails to load the file <cpu>-<os>/vgpreload_<tool name>.so when the tool name contains a hypen. I found out the above by trying to use the name exp-drd for my data race detection tool, which did not work. All regression tests passed however after I switched to the name exp_drd. Bart. |
|
From: Julian S. <js...@ac...> - 2007-11-17 18:37:33
|
On Saturday 17 November 2007 13:51, Bart Van Assche wrote: > I propose to use the prefix exp_ instead of exp-. My motivation is as > follows: * The rule all-local from the top-level Makefile.am (which > installs tool executable and redirections into the .in_place directory) > fails for tools which have a hyphen in their name. > * The core fails to load the file <cpu>-<os>/vgpreload_<tool name>.so > when the tool name contains a hypen. I just fixed the build system for in-place runs to handle this case. (r7174). J |