|
From: Philippe W. <phi...@sk...> - 2011-10-19 21:29:13
|
Have a naming convention for these files looks good.
> I propose to name exp files like so:
>
> BASE.STREAM.exp[-PLATFORM[-FAILCODE][-REASON]]
In the above, we have both . and - as separator, while - is also used
either in the platform (e.g. x86-linux) or in the test name (e.g. threaded-fork.stderr.exp).
=> It might be easier for possible automatic processing and/or ease of wildcards
to systematically use the . to separate the pieces.
Wouldn't a more flexible format be needed ?
With the above it is e.g.
mandatory to give the platform for a kfail.
and if multiple platforms share the same output file, then we need
two (or more) identical files.
So, maybe something like:
BASE.STREAM.exp[.[PLATFORM][.FAILCODE[.REASON]]]
and then in PLATFORM, allows an "or" of platforms (with an _ to separate them ?)
An empty PLATFORM (i.e. two successive . characters) would match any
platform. Not clear to me if you expect that the REASON can only be given
when a FAILCODE is given. The above assumes that.
It might also be better to have a proposal which keeps the "'fixed' file extension parts'
at the end to have e.g. *.exp
*.stdout.exp
matching
(otherwise, you need to always have the double of wildcards, e.g; *.exp *.exp.*).
Not clear however in which order to put what at the end.
> PLATFORM := x86 | x86-linux | x86-darwin | amd64 | arm | ppc32 | ppc64 |
> s390x
...
> The matching in vg_regtest would start with the most specialized exp
> file towards the generic one and it would stop as soon as a match is
> obtained. E.g. when looking for a stderr exp file for foo.vgtest while
> running on x86 Linux, vg_regtest would attempt to match in this order:
>
> foo.stderr.exp-x86-linux.*
> foo.stderr.exp-x86-*
> foo.stderr.exp-32bit-*
32bit is not one of the PLATFORM values above. I guess such 'generic' platforms
is however needed. vg_regtest will have the knowledge to translate a specific
platform in one or more of such generic platforms.
> foo.stderr.exp-*
I do not understand the matching logic and how you will avoid false positive.
This last line will e.g. match foo.stderr:exp-ARM.
And on linux, foo.stderr.exp-x86-* will match foo.stderr.exp-x86-darwin
> There is still a possibility for false positives in the case where
> there are several exp files that differ only in the REASON code. To
> handle that properly we would have to formalize the REASON code.
> I have no plans to do that as this happens rarely.
I am not sure to understand the concept of false positive you are looking at
(mentionned here, but also at the begin of the proposal).
I understand the one at the beginning of the mail i.e. that matching any file can
cause that a regression for a platform matches a "kfail" of another platform, and so
is considered "normal expected failure" while it is in fact a brand new regression.
I do not understand this second "false positive" concept:
Imagine that I have a test which (for bad reasons, but let's assume not easy to fix)
can give randomly two outputs:
foo.stderr.exp-x86-linux-kfail-notfixablebehaviour1
foo.stderr.exp-x86-linux-kfail-notfixablebehaviour2
The above only differs in the REASON code. Why/how can this give a false positive ?
Philippe
|