|
From: Florian K. <br...@ac...> - 2011-10-19 16:01:01
|
Hi, I'd like to formalize the names that we give to the .exp files in the regression test buckets. There are at least three reasons. First, when vg_regtest matches the .out file as obtained by running the test against the existing exp files it will report success if *any* of the existing exp files matches. That is the wrong thing because it can lead to false positives (claiming that a test passes where in fact it does not). In the future, if we're running on, say, ARM and there is an ARM-specific exp file then vg_regtest should only compare against that one and not any others. To do that we must be able to extract the architecture from the file name. Secondly, we have currently a bunch of platform-specific exp files that contain a result that is actually incorrect. E.g. the line numbers in a backtrace are off-by-one. vg_regtest will report success because the actual result matches the incorrect exp file and the issue is hidden. I've begun to name these files by appending -kfail to the name. With this notation I'm piggy-backing on notation invented for POSIX 1003.3 conformance testing. See here: http://www.delorie.com/gnu/docs/dejagnu/dejagnu_6.html This enables us to change vg_regtest to summarize how many testcases passed and how many kfail'ed. Thirdly, consistency. Always a good thing. And simplifies vg_regtest's file name matching. I propose to name exp files like so: BASE.STREAM.exp[-PLATFORM[-FAILCODE][-REASON]] Stuff written in all-caps indicates a syntactic variable, [...] is something optional and | means 'or'. All other characters stand for themselves. BASE := basename of the test; e.g. for foo.vgtest BASE would be foo STREAM := stdout | stderr | stdoutB | stderrB PLATFORM := x86 | x86-linux | x86-darwin | amd64 | arm | ppc32 | ppc64 | s390x FAILCODE := kfail REASON := any character string; needed e.g. if we need more than one exp file for a given platform As usual, matching is greedy. E.g. foo.exp-x86-linux means PLATFORM=x86-linux. It does not mean, PLATFORM=x86, REASON=linux The matching in vg_regtest would start with the most specialized exp file towards the generic one and it would stop as soon as a match is obtained. E.g. when looking for a stderr exp file for foo.vgtest while running on x86 Linux, vg_regtest would attempt to match in this order: foo.stderr.exp-x86-linux.* foo.stderr.exp-x86-* foo.stderr.exp-32bit-* foo.stderr.exp-* There is still a possibility for false positives in the case where there are several exp files that differ only in the REASON code. To handle that properly we would have to formalize the REASON code. I have no plans to do that as this happens rarely. Does anybody see a problem with this approach? Comments welcome. Florian |