|
From: <sv...@va...> - 2007-11-02 21:44:02
|
Author: njn Date: 2007-11-02 21:44:02 +0000 (Fri, 02 Nov 2007) New Revision: 7084 Log: Update todo notes Modified: trunk/massif/ms_main.c Modified: trunk/massif/ms_main.c =================================================================== --- trunk/massif/ms_main.c 2007-11-02 20:47:53 UTC (rev 7083) +++ trunk/massif/ms_main.c 2007-11-02 21:44:02 UTC (rev 7084) @@ -36,7 +36,25 @@ // documentation // - write documentation // - make --threshold and --peak-inaccuracy fractional -// - do filename properly, clean up Valgrind-wide log file naming mess +// - do filename properly, clean up Valgrind-wide log file naming mess. +// Expected behaviour: +// - Main log file: +// default --> stderr +// --log-file=X --> X.<pid> +// --log-file-exactly=X --> X +// --log-file-qualifier=QUAL --> [error] +// --log-file=X --log-file-qualifier=QUAL --> X.$QUAL +// - Massif out file: +// default --> cachegrind.out.<pid> +// --cg-out-file=X --> X.<pid> +// --cg-out-file-exactly=X --> X +// --cg-out-file-qualifier=QUAL --> [error] +// --cg-out-file=X --cg-out-file-qualifier=QUAL --> X.$QUAL +// - Likewise for Cachegrind, but with --cg-out/cg.out. +// - And in cg_annotate, remove the --<pid> option. +// - Likewise for Callgrind, but with --cl-out/cl.out (?) +// - And don't create .1, .2 etc suffixed files. +// // - currently recording asked-for sizes of heap blocks, not actual sizes. // Should add the difference to heap-admin, and change heap-admin name to // something else (heap-extra?). |
|
[Valgrind-developers] Consistent naming options for output [was:
Re: valgrind: r7084 - trunk/massif]
From: Josef W. <Jos...@gm...> - 2007-11-02 23:29:10
|
On Friday 02 November 2007, sv...@va... wrote: > Author: njn > +// - Massif out file: > +// default --> cachegrind.out.<pid> > +// --cg-out-file=X --> X.<pid> > +// --cg-out-file-exactly=X --> X > +// --cg-out-file-qualifier=QUAL --> [error] > +// --cg-out-file=X --cg-out-file-qualifier=QUAL --> X.$QUAL > +// - Likewise for Cachegrind, but with --cg-out/cg.out. > +// - And in cg_annotate, remove the --<pid> option. > +// - Likewise for Callgrind, but with --cl-out/cl.out (?) > +// - And don't create .1, .2 etc suffixed files. Hi Nick, cool, that you want to solve this mess in a consistent way. I am all for it. I am just not sure if the above is flexible enough. Eg. in callgrind, I can have output triggered from different threads, as well as multiple outputs. I think that allowing for a format pattern would be a good way. In this format pattern, we could allow for sequences to be substituted: %p <pid> of running process %t <thread> producing this output %T tool name %c counter to be incremented on each output I do not understand the benefit of QUAL above. Do I understand correctly, and you specify the name of an environment variable here that is substituted in the file name? Why can you not just say --cg-out-file=X.$QUAL (ie. the substitution is done by the shell at exec time)? With %T, we could generalize the options among the tools, e.g. by just using "--out-file=<pattern>". However. This does not really work for callgrind :-( To get the current behavior of callgrind, there is the need for four (!) such patterns for different contexts: 1) termination without separate counters per thread: %T.out.%p 2) termination with separate counters per thread: %T.out.%p-%t 3) dumping in the middle without seperate c.p.t. : %T.out.%p.%c 4) " with " : %T.out.%p.%c-%t I am open for any suggestions to unify this mess into just one pattern. The bad thing is that KCachegrind depends on this to some extend: When running KCachegrind on the generated output from termination, it searches for any further outputs with suffixes. However, I want to get rid of this because it is quite a confusing behavior. Josef |
|
From: Nicholas N. <nj...@cs...> - 2007-11-09 22:52:28
|
On Sat, 3 Nov 2007, Josef Weidendorfer wrote: > I do not understand the benefit of QUAL above. Do I understand correctly, > and you specify the name of an environment variable here that is substituted > in the file name? Why can you not just say --cg-out-file=X.$QUAL (ie. the > substitution is done by the shell at exec time)? It's somehow important for MPI programs, where you want every different process's output to go to a different file. I don't know the details. > With %T, we could generalize the options among the tools, e.g. by > just using "--out-file=<pattern>". > However. This does not really work for callgrind :-( > To get the current behavior of callgrind, there is the need for > four (!) such patterns for different contexts: > 1) termination without separate counters per thread: %T.out.%p > 2) termination with separate counters per thread: %T.out.%p-%t > 3) dumping in the middle without seperate c.p.t. : %T.out.%p.%c > 4) " with " : %T.out.%p.%c-%t > > I am open for any suggestions to unify this mess into just one pattern. A format string seems like a good way to do it. But it's not obvious how to do it for Callgrind. Hmm. Nick |
|
From: Ashley P. <api...@co...> - 2007-11-12 20:20:22
|
On Sat, 2007-11-10 at 09:52 +1100, Nicholas Nethercote wrote: > On Sat, 3 Nov 2007, Josef Weidendorfer wrote: > > > I do not understand the benefit of QUAL above. Do I understand correctly, > > and you specify the name of an environment variable here that is substituted > > in the file name? Why can you not just say --cg-out-file=X.$QUAL (ie. the > > substitution is done by the shell at exec time)? > > It's somehow important for MPI programs, where you want every different > process's output to go to a different file. I don't know the details. Typically with parallel applications you launch many copies of the same program with the same command line parameters, the MPI implementation itself numbers the processes from 0 to N-1 given N processes, this is known as the MPI "rank" of a process. Most job schedulers used for launching parallel jobs export the rank as a environment variable so any wrapper script, such as valgrind, can use this to name log files et al. Not having this in cachegrind frequently causes problems, I don't think cachegrind supports --log-file-exactly currently which means I can't even use a wrapper shell script to deference the variable. I have a patch somewhere to add the qualifier to the name of any core file generated, I could dig it out if you think it would help although I if possible I think it would be better so abstract all this out so all the file naming code is handled centrally and all behaves the same way. Ashley, |
|
From: Nicholas N. <nj...@cs...> - 2007-11-12 21:08:43
|
On Mon, 12 Nov 2007, Ashley Pittman wrote: > Typically with parallel applications you launch many copies of the same > program with the same command line parameters, the MPI implementation > itself numbers the processes from 0 to N-1 given N processes, this is > known as the MPI "rank" of a process. Most job schedulers used for > launching parallel jobs export the rank as a environment variable so any > wrapper script, such as valgrind, can use this to name log files et al. > > Not having this in cachegrind frequently causes problems, I don't think > cachegrind supports --log-file-exactly currently which means I can't > even use a wrapper shell script to deference the variable. $QUAL support has been added in the trunk, although it's inconsistent at the moment (i.e. the naming scheme is inconsistent). It'll be fixed for 3.3.0. N |
|
From: Josef W. <Jos...@gm...> - 2007-11-14 20:55:15
|
On Monday 12 November 2007, Nicholas Nethercote wrote:
> On Mon, 12 Nov 2007, Ashley Pittman wrote:
>
> > Typically with parallel applications you launch many copies of the same
> > program with the same command line parameters, the MPI implementation
> > itself numbers the processes from 0 to N-1 given N processes, this is
> > known as the MPI "rank" of a process. Most job schedulers used for
> > launching parallel jobs export the rank as a environment variable so any
> > wrapper script, such as valgrind, can use this to name log files et al.
> >
> > Not having this in cachegrind frequently causes problems, I don't think
> > cachegrind supports --log-file-exactly currently which means I can't
> > even use a wrapper shell script to deference the variable.
>
> $QUAL support has been added in the trunk, although it's inconsistent at the
> moment (i.e. the naming scheme is inconsistent). It'll be fixed for 3.3.0.
Can we solve this QUAL issue also via format string, e.g. via "...%{QUAL}...",
such that %{QUAL} is replaced by Valgrind with $QUAL from the current
environment?
I thought a little bit about the problem with callgrind.
IMHO I need multiple format strings for different scenarios, but
the format strings could refer to substitution results of each other.
E.g. it would be nice if the consistent output name specified with "--out-file=..."
could be refered in further arbitrary format strings as the tool requires, e.g.
as "...%o..." (for _o_utput file). Then, e.g. the default for "termination
with separate counters per thread" would be "%o-%t".
Josef
>
> N
>
|
|
From: Nicholas N. <nj...@cs...> - 2007-11-15 04:54:54
|
On Wed, 14 Nov 2007, Josef Weidendorfer wrote:
> Can we solve this QUAL issue also via format string, e.g. via "...%{QUAL}...",
> such that %{QUAL} is replaced by Valgrind with $QUAL from the current
> environment?
Yes, Julian and I had come to the same conclusion.
> I thought a little bit about the problem with callgrind.
> IMHO I need multiple format strings for different scenarios, but
> the format strings could refer to substitution results of each other.
> E.g. it would be nice if the consistent output name specified with "--out-file=..."
> could be refered in further arbitrary format strings as the tool requires, e.g.
> as "...%o..." (for _o_utput file). Then, e.g. the default for "termination
> with separate counters per thread" would be "%o-%t".
I didn't really follow that, but it seems like Callgrind produces multiple
output files. They all have a common prefix, and then each one has a
different suffix. When it comes to processing them, presumably you give
callgrind_annotate/KCachegrind the prefix, and then they can find all the
extra files because they know what the suffixes look like.
In other words, it makes sense for a user to specify the prefix, but not the
suffix, because the processing tools depend on them having a particular
form.
If I'm right, then I think we only need:
%p process id
%q{QUAL} environment variable contents
%T for toolname isn't necessary, nor are %t or %c because they're not
controllable by the user.
Overall, we'd have a --log-file option for normal Valgrind output (ie. what
normally goes to stderr), and then for each tool that writes an output file
(Cachegrind, Callgrind, Massif) there would be an option like --massif-file.
In the Callgrind case it would be --callgrind-file-prefix.
Does this make sense?
Nick
|
|
From: Josef W. <Jos...@gm...> - 2007-11-15 10:01:05
|
On Thursday 15 November 2007, Nicholas Nethercote wrote:
> I didn't really follow that, but it seems like Callgrind produces multiple
> output files. They all have a common prefix, and then each one has a
> different suffix. When it comes to processing them, presumably you give
> callgrind_annotate/KCachegrind the prefix, and then they can find all the
> extra files because they know what the suffixes look like.
Yes, that's right in the case of KCachegrind.
I assume there could be other tools in the future which produce multiple
output files. It is reasonable to expect that they all will be fine
with a user settable prefix part and a tool controlled suffix.
> In other words, it makes sense for a user to specify the prefix, but not the
> suffix, because the processing tools depend on them having a particular
> form.
>
> If I'm right, then I think we only need:
>
> %p process id
> %q{QUAL} environment variable contents
>
> %T for toolname isn't necessary, nor are %t or %c because they're not
> controllable by the user.
Hm. %t could make sense with a tool which has an output per thread.
But as there currently is no such tool, there is no need.
(for callgrind, the thread is part of the suffix which the user does not
have the need to control, so there is no need for %t there, too).
> Overall, we'd have a --log-file option for normal Valgrind output (ie. what
> normally goes to stderr), and then for each tool that writes an output file
> (Cachegrind, Callgrind, Massif) there would be an option like --massif-file.
> In the Callgrind case it would be --callgrind-file-prefix.
Do we really want the tool name in the option? I would vote for one global
option "--output-file=..." to specify an output file name/pattern, and the tool
is free to add suffixes. This makes it easier for the user to remember
* the option name and
* the fact that this option allows for pattern specification.
I currently have the "--base=<prefix>" option in callgrind which does _not_ allow
for patterns. I will deprecate it, but should support it for some time for
backwards compatibility. I think it really would confuse the user if I additionally
provide a "--callgrind-file-prefix" which allows for patterns...
Josef
>
> Does this make sense?
>
> Nick
>
|
|
From: Nicholas N. <nj...@cs...> - 2007-11-15 11:07:28
|
On Thu, 15 Nov 2007, Josef Weidendorfer wrote: >> Overall, we'd have a --log-file option for normal Valgrind output (ie. what >> normally goes to stderr), and then for each tool that writes an output file >> (Cachegrind, Callgrind, Massif) there would be an option like --massif-file. >> In the Callgrind case it would be --callgrind-file-prefix. > > Do we really want the tool name in the option? I would vote for one global > option "--output-file=..." to specify an output file name/pattern, and the tool > is free to add suffixes. This makes it easier for the user to remember > * the option name and > * the fact that this option allows for pattern specification. An example: for Cachegrind, there is the "log output", ie. what normally is printed to stderr, and there is the "file output", ie. what goes in cachegrind.out.*. You need to be able to control the file names of these two separately. So a single global options won't suffice. Nick |
|
From: Josef W. <Jos...@gm...> - 2007-11-15 14:02:41
|
On Thursday 15 November 2007, Nicholas Nethercote wrote: > On Thu, 15 Nov 2007, Josef Weidendorfer wrote: > > >> Overall, we'd have a --log-file option for normal Valgrind output (ie. what > >> normally goes to stderr), and then for each tool that writes an output file > >> (Cachegrind, Callgrind, Massif) there would be an option like --massif-file. > >> In the Callgrind case it would be --callgrind-file-prefix. > > > > Do we really want the tool name in the option? I would vote for one global > > option "--output-file=..." to specify an output file name/pattern, and the tool > > is free to add suffixes. This makes it easier for the user to remember > > * the option name and > > * the fact that this option allows for pattern specification. > > An example: for Cachegrind, there is the "log output", ie. what normally is > printed to stderr, and there is the "file output", ie. what goes in > cachegrind.out.*. You need to be able to control the file names of these > two separately. So a single global options won't suffice. The "global single option" was meant to be about the output file only. Of course there also is a log output that already can be redirected with "--log-file". But we currently do not have a per-tool option for the log output of each tool. Why do we want different options to specify the output file of each tool then? Josef > > Nick > |
|
From: Nicholas N. <nj...@cs...> - 2007-11-15 21:43:26
|
On Thu, 15 Nov 2007, Josef Weidendorfer wrote: > The "global single option" was meant to be about the output file only. > Of course there also is a log output that already can be redirected > with "--log-file". But we currently do not have a per-tool option for the log > output of each tool. Why do we want different options to specify the output > file of each tool then? Ok, I think you're suggesting we have just two options, something like: --log-file: for logging output, eg. what normally goes to stderr --out-file: for tool-specific file output, eg. Cachegrind/Massif I find --out-file confusing. The difference with --log-file is not immediately clear. Also, what about tools that don't create a file, like Memcheck -- is --out-file ignored? Should it cause an abort if it's used? I think having options that all tools must implement is not a good idea. We already have some (eg. -v) but I'd like to avoid more. In comparison, if we have a --cachegrind-out-file option, it's very clear what it does, particularly if the default name is "cachegrind.out.<pid>". Nick |
|
From: Josef W. <Jos...@gm...> - 2007-11-16 11:05:56
|
On Thursday 15 November 2007, Nicholas Nethercote wrote: > On Thu, 15 Nov 2007, Josef Weidendorfer wrote: > > > The "global single option" was meant to be about the output file only. > > Of course there also is a log output that already can be redirected > > with "--log-file". But we currently do not have a per-tool option for the log > > output of each tool. Why do we want different options to specify the output > > file of each tool then? > > Ok, I think you're suggesting we have just two options, something like: > > --log-file: for logging output, eg. what normally goes to stderr > > --out-file: for tool-specific file output, eg. Cachegrind/Massif Yes, that was the idea. > I find --out-file confusing. The difference with --log-file is not > immediately clear. Which difference? Do you think that people could confuse these two options? > Also, what about tools that don't create a file, like > Memcheck -- is --out-file ignored? Should it cause an abort if it's used? > I think having options that all tools must implement is not a good idea. We > already have some (eg. -v) but I'd like to avoid more. It just would be handled the same as other tool-specific options. Ie. it only would be a convention among tools, not technically enforced. > In comparison, if we have a --cachegrind-out-file option, it's very clear > what it does, particularly if the default name is "cachegrind.out.<pid>". I am not really settled about "--out-file". I just think that options for similar functions in different tools should show some consistency. So I am really fine with an option "--<toolname>-out-file=..." with support for patterns (like %p). So I would change the according callgrind option to "--callgrind-out-file=...", independent of the fact that callgrind could produce further files with additional suffix. Is this OK? Josef > > Nick > |
|
From: Nicholas N. <nj...@cs...> - 2007-11-17 00:13:41
|
On Fri, 16 Nov 2007, Josef Weidendorfer wrote: > I am not really settled about "--out-file". I just think that options for > similar functions in different tools should show some consistency. > So I am really fine with an option "--<toolname>-out-file=..." with support > for patterns (like %p). > > So I would change the according callgrind option to "--callgrind-out-file=...", > independent of the fact that callgrind could produce further files with > additional suffix. > > Is this OK? That sounds ok. In summary: --log-file=<pattern> --cachegrind-out-file=<pattern> --massif-out-file=<pattern> --callgrind-out-file=<pattern> The latter could also be --callgrind-out-prefix, but I'm ok with --callgrind-out-file. We could also shorten the options names, eg. --cg-out-file, but I'm not too fussed about that. Nick |
|
From: Josef W. <Jos...@gm...> - 2007-11-17 00:52:23
|
On Saturday 17 November 2007, Nicholas Nethercote wrote: > On Fri, 16 Nov 2007, Josef Weidendorfer wrote: > > > I am not really settled about "--out-file". I just think that options for > > similar functions in different tools should show some consistency. > > So I am really fine with an option "--<toolname>-out-file=..." with support > > for patterns (like %p). > > > > So I would change the according callgrind option to "--callgrind-out-file=...", > > independent of the fact that callgrind could produce further files with > > additional suffix. > > > > Is this OK? > > That sounds ok. In summary: > > --log-file=<pattern> > > --cachegrind-out-file=<pattern> > > --massif-out-file=<pattern> > > --callgrind-out-file=<pattern> > > The latter could also be --callgrind-out-prefix, but I'm ok with > --callgrind-out-file. The thing is that for normal use, it is the not a prefix, but the output file name. And even in other cases with multiple files, one will have exactly this name. So I think that should be fine. > We could also shorten the options names, eg. --cg-out-file, but I'm not too > fussed about that. I think the option will not be used to often; so a long name should be ok. The nice thing is that the long name matches quite fine the default output file name (for all of cachegrind, callgrind, massif). Josef > > Nick > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2005. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > Valgrind-developers mailing list > Val...@li... > https://lists.sourceforge.net/lists/listinfo/valgrind-developers > |
|
From: Julian S. <js...@ac...> - 2007-11-17 02:41:37
|
On Saturday 17 November 2007 01:13, Nicholas Nethercote wrote:
> That sounds ok. In summary:
>
> --log-file=<pattern>
>
> --cachegrind-out-file=<pattern>
>
> --massif-out-file=<pattern>
>
> --callgrind-out-file=<pattern>
That sounds good to me. With patterns
%p process ID
%q(VAR) contents of $VAR
I would prefer to use ( ) instead of { } around VAR. { and }
interact really badly with shells and are generally a pain to
work with.
J
|
|
From: Oswald B. <os...@kd...> - 2007-11-17 09:23:44
|
On Sat, Nov 17, 2007 at 03:41:24AM +0100, Julian Seward wrote:
> On Saturday 17 November 2007 01:13, Nicholas Nethercote wrote:
> > That sounds ok. In summary:
> >
> > --log-file=<pattern>
>
> That sounds good to me. With patterns
>
> %p process ID
> %q(VAR) contents of $VAR
>
> I would prefer to use ( ) instead of { } around VAR. { and }
> interact really badly with shells and are generally a pain to
> work with.
>
oh, and ( ) do not interact badly with shells, huh? ;)
apart from that, { } is perfectly fine as long as there is no comma in
between.
last but not least, { } just feels more "natural" in scripting context.
only makefiles use $() for variable expansion, but there it has some
more meanings, too.
--
Hi! I'm a .signature virus! Copy me into your ~/.signature, please!
--
Chaos, panic, and disorder - my work here is done.
|
|
From: Josef W. <Jos...@gm...> - 2007-11-17 14:22:30
|
On Saturday 17 November 2007, Julian Seward wrote:
> I would prefer to use ( ) instead of { } around VAR. { and }
> interact really badly with shells and are generally a pain to
> work with.
Hmm.. Is ( and ) really better?
weidendo@linux:~> echo %q{foo}
%q{foo}
weidendo@linux:~> echo %q(foo)
bash: syntax error near unexpected token `('
weidendo@linux:~>
But I do not really have any preference.
Josef
>
> J
>
|
|
From: Julian S. <js...@ac...> - 2007-11-17 18:41:42
|
> Hmm.. Is ( and ) really better?
Hmm .. no. It seems like the evidence is not in my favour :-)
> weidendo@linux:~> echo %q{foo}
> %q{foo}
> weidendo@linux:~> echo %q(foo)
> bash: syntax error near unexpected token `('
> weidendo@linux:~>
That's in bash. In csh:
> echo %q{foo}
%qfoo
> echo %q(foo)
Badly placed ()'s.
So in csh both are borked, but ( ) is more borked than { }.
Can I change my vote back to { } please?
J
|
|
From: Nicholas N. <nj...@cs...> - 2007-11-17 23:58:50
|
On Sat, 17 Nov 2007, Julian Seward wrote:
>> weidendo@linux:~> echo %q{foo}
>> %q{foo}
>> weidendo@linux:~> echo %q(foo)
>> bash: syntax error near unexpected token `('
>> weidendo@linux:~>
>
> That's in bash. In csh:
>
> > echo %q{foo}
> %qfoo
> > echo %q(foo)
> Badly placed ()'s.
>
> So in csh both are borked, but ( ) is more borked than { }.
>
> Can I change my vote back to { } please?
Another possibility: %qVAR
N
|
|
From: Julian S. <js...@ac...> - 2007-11-18 00:38:59
|
> Another possibility: %qVAR That would be good, in the sense that the shell can't screw it up. Problem is there's no way to know where the env var ends and we move back to ordinary command line text. Maybe %qVAR% or %qVAR%q ? Both are ugly and nonintuitive, though. J |
|
From: Ashley P. <api...@co...> - 2007-11-18 17:59:41
|
On Sun, 2007-11-18 at 01:38 +0100, Julian Seward wrote: > > Another possibility: %qVAR > > That would be good, in the sense that the shell can't screw it up. > Problem is there's no way to know where the env var ends and we move > back to ordinary command line text. > > Maybe %qVAR% or %qVAR%q ? Both are ugly and nonintuitive, though. I was going to say go with %q%VAR% but I'm not sure that qualifier is the right name to use, how about using %env%VAR% for environment stuff and %pid% for the pid? Ashley, |
|
From: Julian S. <js...@ac...> - 2007-11-18 00:42:47
|
On Sunday 18 November 2007 01:38, Julian Seward wrote: > > Another possibility: %qVAR > > That would be good, in the sense that the shell can't screw it up. s/can't/can/g Duh. J |
|
From: Julian S. <js...@ac...> - 2007-11-18 00:48:53
|
On Sunday 18 November 2007 01:42, Julian Seward wrote: > On Sunday 18 November 2007 01:38, Julian Seward wrote: > > > Another possibility: %qVAR > > > > That would be good, in the sense that the shell can't screw it up. > > s/can't/can/g Now I'm really feeling stupid. The first version was correct. J |