|
From: Ashley P. <as...@pi...> - 2009-08-13 15:14:49
|
Hi,
I've been testing the xml output of valgrind and the new options since
the merge. I've recently returned from holiday and have read several
discussions on the mailing lists but thought it might be appropriate to
start a new thread.
Points I've spotted are:
a) Why do we still need --xml=yes, isn't it implied by
--xml-file=<file>? It would appear the --xml=yes option can be dropped
completely, if used without a XML output destination it aborts anyway.
In addition the error is reported to the user twice, in different
formats.
ashley@alpha:/mnt/memfs/val-install/bin$ ./valgrind --xml=yes ls
==18750== --xml=yes has been specified, but there is no XML output
==18750== destination. You must specify an XML output destination
==18750== using --xml-fd=, --xml-file= or --xml=socket=.
valgrind: Bad option '--xml=yes, but no XML destination specified';
aborting.
valgrind: Use --help for more information.
ashley@alpha:/mnt/memfs/val-install/bin$
b) If I do specify --xml-file=<file> but not --xml=yes I get partial
output in the xml file.
ashley@alpha:/mnt/memfs/val-install/bin$ ./valgrind --xml-file=x
--log-file=l ls
callgrind_annotate cg_annotate l no_op_client_for_valgrind
valgrind-listener
callgrind_control cg_merge ms_print valgrind x
ashley@alpha:/mnt/memfs/val-install/bin$ cat x
<errorcounts>
</errorcounts>
ashley@alpha:/mnt/memfs/val-install/bin$
c) It doesn't seem to be possible to get both the XML output and the
standard output. I realise it wasn't before but when I read on the list
that it was it struck me as a good idea. More on this later.
d) I like the idea of having separate streams for XML and serious error
messages, ideally these would also be encapsulated in the xml but in
practice they are rare enough to get away without this. I'd like to use
--xml-socket= --log-socket= to catch the output of multiple processes
simultaneously however there would be no way to identify which process
any output on a non-xml socket came from.
e) There is no way to get a "qualifier" over a xml socket.
f) Using log file qualifiers with XML results in the xml for the
qualifier being put to stdout
ashley@alpha:/mnt/memfs/val-install/bin$ ./valgrind --xml-file=x.%q{ID}
--xml=yes ls
==18777== <logfilequalifier> <var>ID</var> <value>0</value>
</logfilequalifier>
callgrind_annotate cg_annotate l no_op_client_for_valgrind
valgrind-listener
callgrind_control cg_merge ms_print valgrind x.0
g) The "Command:" <line> in the xml preamble is superfluous.
All these tests are preformed with the head (10800M), the diffs all
relate to VPATH building.
With regard to c) one of the tests I did before was to try to
re-construct the normal output from the xml output (it's a very good way
of testing the xml parsing), previously I've always just run the
application twice, once in each mode however this limits the type of
application you can test.
I was wondering if would be possible to change the tests to use
something like this, have the test suite check that 1) the xml output
can be used to re-create the normal output, byte-for-btye and 2) the xml
output matches the expected xml for that test. For this you'd need to
re-write the "expected output" files in some new kind of xml but that
shouldn't be too hard, all it would need to say was which fields of
output needed to match (function names, error types) and which didn't
(line numbers).
Ashley,
--
Ashley Pittman, Bath, UK.
Padb - A parallel job inspection tool for cluster computing
http://padb.pittman.org.uk
|
|
From: Nicholas N. <n.n...@gm...> - 2009-08-14 22:49:10
|
On Fri, Aug 14, 2009 at 1:15 AM, Ashley Pittman<as...@pi...> wrote:
>
> Points I've spotted are:
>
> a) Why do we still need --xml=yes, isn't it implied by
> --xml-file=<file>? It would appear the --xml=yes option can be dropped
> completely, if used without a XML output destination it aborts anyway.
>
> b) If I do specify --xml-file=<file> but not --xml=yes I get partial
> output in the xml file.
>
> <errorcounts>
> </errorcounts>
The small amount of output in (b) looks like a bug.
But I agree that the option behaviour is sub-optimal and inconsistent,
esp. that you can specify --xml-file without --xml=yes and it'll
continue.
I'd suggest:
- Allow --xml=yes for backward compatibility. If it's specified, then
--xml-{file,socket,fd} also has to be specified
- If --xml-{file,socket,fd} is given, --xml=yes doesn't have to be,
and you get full XML output.
Nick
|
|
From: Ashley P. <as...@pi...> - 2009-08-15 09:41:52
|
On Sat, 2009-08-15 at 08:48 +1000, Nicholas Nethercote wrote:
> On Fri, Aug 14, 2009 at 1:15 AM, Ashley Pittman<as...@pi...> wrote:
> > a) Why do we still need --xml=yes, isn't it implied by
> > --xml-file=<file>? It would appear the --xml=yes option can be dropped
> > completely, if used without a XML output destination it aborts anyway.
> >
> > b) If I do specify --xml-file=<file> but not --xml=yes I get partial
> > output in the xml file.
> >
> > <errorcounts>
> > </errorcounts>
>
> The small amount of output in (b) looks like a bug.
Yes. Interestingly enough this output was both the two xml tags and a
number of new-lines. Point (f) appears to be a simple bug as well, the
log file qualifier goes to stdout instead of the xml log file.
> But I agree that the option behaviour is sub-optimal and inconsistent,
> esp. that you can specify --xml-file without --xml=yes and it'll
> continue.
>
> I'd suggest:
>
> - Allow --xml=yes for backward compatibility. If it's specified, then
> --xml-{file,socket,fd} also has to be specified
>
> - If --xml-{file,socket,fd} is given, --xml=yes doesn't have to be,
> and you get full XML output.
That would seem sensible to me, the error message if you specify only
--xml=yes says what extra steps you need to take to enable xml so should
be sufficient, I can see the reason for not wanting to drop the option
completely.
Did you have any thoughts on the other points I mentioned, either
getting both sets of output and ideally a way of getting a qualifier in
over the --{log,file}-socket options.
Ashley,
--
Ashley Pittman, Bath, UK.
Padb - A parallel job inspection tool for cluster computing
http://padb.pittman.org.uk
|
|
From: Nicholas N. <n.n...@gm...> - 2009-08-15 11:52:19
|
On Sat, Aug 15, 2009 at 7:42 PM, Ashley Pittman<as...@pi...> wrote:
>
> Did you have any thoughts on the other points I mentioned, either
> getting both sets of output and ideally a way of getting a qualifier in
> over the --{log,file}-socket options.
I don't know much about the qualifier issue. As for both sets of
output, it's conceptually nice but does it have a practical use? (I
think it would invalidate Julian's use case of looking in the non-XML
text output to look for critical errors.) Having said that, the
message printing code currently is ugly, there's lots of ad hoc tests
all over the place to decide whether to print each message as text or
XML or both, it would be great to have a simpler and more consistent
story for all that.
I very much like the idea of testing the XML by converting it back to
text and comparing with the text output, but it would be too much to
do it for every regtest, I think. Is there a simple GPL-licensed XML
parser that could be the starting point for this, preferably in C or
Perl? Presumably.
Nick
|
|
From: Ashley P. <as...@pi...> - 2009-08-15 12:23:26
|
On Sat, 2009-08-15 at 21:52 +1000, Nicholas Nethercote wrote:
> On Sat, Aug 15, 2009 at 7:42 PM, Ashley Pittman<as...@pi...> wrote:
> >
> > Did you have any thoughts on the other points I mentioned, either
> > getting both sets of output and ideally a way of getting a qualifier in
> > over the --{log,file}-socket options.
>
> I don't know much about the qualifier issue. As for both sets of
> output, it's conceptually nice but does it have a practical use? (I
> think it would invalidate Julian's use case of looking in the non-XML
> text output to look for critical errors.) Having said that, the
> message printing code currently is ugly, there's lots of ad hoc tests
> all over the place to decide whether to print each message as text or
> XML or both, it would be great to have a simpler and more consistent
> story for all that.
I work in HPC (High performance computing) where applications are
written in MPI and run in parallel. Ideally I'd like to start a
valgrind-listener type process, pass the socket id to multiple valgrind
instances simultaneously and then report errors as the sum of the
instances rather than on a per instance basis.
As you say this does make Julian's use case invalid which is unfortunate
as it's a very useful one, perhaps it would be possible to print the
pre-amble and qualifiers only if using --log-socket and then test for
further output as a sign of critical error?
> I very much like the idea of testing the XML by converting it back to
> text and comparing with the text output, but it would be too much to
> do it for every regtest, I think. Is there a simple GPL-licensed XML
> parser that could be the starting point for this, preferably in C or
> Perl? Presumably.
Yes, I've written one in perl. It was to solve the problem above, at
the time I used files with a qualifier and then after the job had
finished loaded all the xml files. I would then re-write the "standard"
valgrind output prefixing each error with a "qualifier prefix" which is
the standard way of specifying processes within HPC. You'd end up with
an extra line for each error, as shown below.
==27271== [0,2-15]
==27271== Use of uninitialised value of size 8
==27271== at 0x187FCEC6: (within /usr/lib/libGLcore.so.180.44)
==27271== by 0x41FF6BF: (within /usr/lib/libGL.so.180.44)
==27271== by 0x41FF6BF: (within /usr/lib/libGL.so.180.44)
It order to test the xml parser and subsequent output code you could run
the program with only one input file without a prefix and it would
simply convert the output as best it could, I even hooked this into the
test suite so the regtests did this automatically and then tested the
resultant output in the normal way, by and large this worked well with a
few specific cases where it didn't.
That was a couple of years ago now, it's written in perl, GPL but will
probably need updating. It uses XML::Simple which is easy to code but
unfortunately quite slow with large XML files, I did make a stab at
re-writing it to use XML::Parser but this was taking more time than I
had available at the time.
Ashley,
--
Ashley Pittman, Bath, UK.
Padb - A parallel job inspection tool for cluster computing
http://padb.pittman.org.uk
|
|
From: Julian S. <js...@ac...> - 2009-08-15 23:08:22
|
Hi Ashley, Thank you for giving this stuff a good testing. I at least fixed the two obvious bugs you found: > b) If I do specify --xml-file=<file> but not --xml=yes I get partial > output in the xml file. and > f) Using log file qualifiers with XML results in the xml for the > qualifier being put to stdout (at least, I think they are fixed). r10823. J |
|
From: Julian S. <js...@ac...> - 2009-08-15 23:16:16
|
Ashley,
I think you have some good points, esp w.r.t. the tagging when using
--xml-socket. I see that the lack of it gets in your way.
Can I suggest you open a bug report, to put this stuff on? We're trying
to use bugzilla more actively, and it'll stop this all getting lost
otherwise.
> Did you have any thoughts on the other points I mentioned, either
> getting both sets of output and ideally a way of getting a qualifier in
> over the --{log,file}-socket options.
Could you show some examples of what you mean? (examples with simple
bits of XML ?) I think I know what you're getting at, but am not sure.
J
|
|
From: Ashley P. <as...@pi...> - 2009-08-16 11:30:34
|
On Sun, 2009-08-16 at 01:25 +0200, Julian Seward wrote:
> Ashley,
>
> I think you have some good points, esp w.r.t. the tagging when using
> --xml-socket. I see that the lack of it gets in your way.
>
> Can I suggest you open a bug report, to put this stuff on? We're trying
> to use bugzilla more actively, and it'll stop this all getting lost
> otherwise.
Ok, I'll open one.
> > Did you have any thoughts on the other points I mentioned, either
> > getting both sets of output and ideally a way of getting a qualifier in
> > over the --{log,file}-socket options.
>
> Could you show some examples of what you mean? (examples with simple
> bits of XML ?) I think I know what you're getting at, but am not sure.
I'll put something together. Even without using it to try and simplify
the testing it would be useful to be able to reproduce the non xml
output from the xml output.
In the mean time all that's needed from Valgrind is the ability to
output both xml and the standard output simultaneously.
Ashley.
--
Ashley Pittman, Bath, UK.
Padb - A parallel job inspection tool for cluster computing
http://padb.pittman.org.uk
|
|
From: Ashley P. <as...@pi...> - 2009-08-16 15:51:06
Attachments:
v-xml.patch
|
On Thu, 2009-08-13 at 16:15 +0100, Ashley Pittman wrote: > > g) The "Command:" <line> in the xml preamble is superfluous. Attached is a patch to fix this issue. Ashley, -- Ashley Pittman, Bath, UK. Padb - A parallel job inspection tool for cluster computing http://padb.pittman.org.uk |
|
From: Ashley P. <as...@pi...> - 2009-08-17 21:41:16
|
On Sun, 2009-08-16 at 01:25 +0200, Julian Seward wrote: > Ashley, > > I think you have some good points, esp w.r.t. the tagging when using > --xml-socket. I see that the lack of it gets in your way. > > Can I suggest you open a bug report, to put this stuff on? We're trying > to use bugzilla more actively, and it'll stop this all getting lost > otherwise. Done, or at least about the identifiers when using sockets. I got the impression from reading the lists that getting both sets of output was possible using the correct options, is this correct or should I file a feature request bug for this as well? Ashley, -- Ashley Pittman, Bath, UK. Padb - A parallel job inspection tool for cluster computing http://padb.pittman.org.uk |
|
From: Nicholas N. <n.n...@gm...> - 2009-08-17 22:27:10
|
On Tue, Aug 18, 2009 at 7:41 AM, Ashley Pittman<as...@pi...> wrote: > > I got the impression from reading the lists that getting both sets of > output was possible using the correct options, is this correct I don't think so. Eg. if you turn XML output on all errors (as in, those where Valgrind complains about your program) only go to the XML channel, AFAIK. Nick |