[Valgrind-developers] RFC: formalize .exp file names

[Florian K. <br...@ac...>]

Hi,

I'd like to formalize the names that we give to the .exp files in
the regression test buckets. There are at least three reasons.

First, when vg_regtest matches the .out file as obtained by running
the test against the existing exp files it will report success if
*any* of the existing exp files matches. That is the wrong thing
because it can lead to false positives (claiming that a test passes
where in fact it does not). In the future, if we're running on,
say, ARM and there is an ARM-specific exp file then vg_regtest should
only compare against that one and not any others. To do that we must
be able to extract the architecture from the file name.

Secondly, we have currently a bunch of platform-specific exp files that
contain a result that is actually incorrect. E.g. the line numbers in a
backtrace are off-by-one. vg_regtest will report success because the
actual result matches the incorrect exp file and the issue is hidden.
I've begun to name these files by appending -kfail to the name. With
this notation I'm piggy-backing on notation invented for POSIX 1003.3
conformance testing.
See here: http://www.delorie.com/gnu/docs/dejagnu/dejagnu_6.html
This enables us to change vg_regtest to summarize how many testcases
passed and how many kfail'ed.

Thirdly, consistency. Always a good thing. And simplifies vg_regtest's
file name matching.

I propose to name exp files like so:

BASE.STREAM.exp[-PLATFORM[-FAILCODE][-REASON]]

Stuff written in all-caps indicates a syntactic variable, [...] is
something optional and | means 'or'. All other characters stand for
themselves.

BASE := basename of the test; e.g. for foo.vgtest BASE would be foo

STREAM := stdout | stderr | stdoutB | stderrB

PLATFORM := x86 | x86-linux | x86-darwin | amd64 | arm | ppc32 | ppc64 |
s390x

FAILCODE := kfail

REASON := any character string; needed e.g. if we need more than one
          exp file for a given platform

As usual, matching is greedy. E.g. foo.exp-x86-linux means
PLATFORM=x86-linux. It does not mean, PLATFORM=x86, REASON=linux

The matching in vg_regtest would start with the most specialized exp
file towards the generic one and it would stop as soon as a match is
obtained. E.g. when looking for a stderr exp file for foo.vgtest while
running on x86 Linux, vg_regtest would attempt to match in this order:

foo.stderr.exp-x86-linux.*
foo.stderr.exp-x86-*
foo.stderr.exp-32bit-*
foo.stderr.exp-*

There is still a possibility for false positives in the case where
there are several exp files that differ only in the REASON code. To
handle that properly we would have to formalize the REASON code.
I have no plans to do that as this happens rarely.

Does anybody see a problem with this approach? Comments welcome.

  Florian