Re: [Valgrind-developers] RFC: formalize .exp file names

[Florian K. <br...@ac...>]

On 10/19/2011 05:29 PM, Philippe Waroquiers wrote:
> Have a naming convention for these files looks good.
> 
>> I propose to name exp files like so:
>>
>> BASE.STREAM.exp[-PLATFORM[-FAILCODE][-REASON]]
> 
> In the above, we have both . and - as separator, while - is also used
> either in the platform (e.g. x86-linux) or in the test name (e.g.
> threaded-fork.stderr.exp).
> => It might be easier for possible automatic processing and/or ease of
> wildcards
> to systematically use the . to separate the pieces.
> 

That is true. It will simplify things and that's good.

> Wouldn't a more flexible format be needed ?

I don't want to go there. As Bart has written in his reply testcases
ought to be written such that they pass everywhere. That will not always
be possible but we should strive towards that goal. Unless of course
we're testing something platform-specific in which case the testcases
should go into a subdirectory.
In essence a platform-specific exp file is an exception and I rather not
come up with a complex scheme to support it.

>  With the above it is e.g.
>      mandatory to give the platform for a kfail.

Yes. I think that can be salvaged. We actually have a few .exp-kfail
files already.

>      and if multiple platforms share the same output file, then we need
>      two (or more) identical files.
> 
> So, maybe something like:
>   BASE.STREAM.exp[.[PLATFORM][.FAILCODE[.REASON]]]
> 
> and then in PLATFORM, allows an "or" of platforms (with an _ to separate
> them ?)
> An empty PLATFORM (i.e. two successive . characters) would match any
> platform. Not clear to me if you expect that the REASON can only be given
> when a FAILCODE is given. The above assumes that.
> 

There are currently only two testcases memcheck/tests/origin5-bz2 and
memcheck/tests/partiallydefinedeq where this would be helpful. Not
enough to get me excited about.

>> vg_regtest would attempt to match in this order:
>>
>> foo.stderr.exp-x86-linux.*
>> foo.stderr.exp-x86-*
>> foo.stderr.exp-32bit-*
> 32bit is not one of the PLATFORM values above. I guess such 'generic'
> platforms
> is however needed. vg_regtest will have the knowledge to translate a
> specific
> platform in one or more of such generic platforms.
> 

Yes, right.

>> foo.stderr.exp-*
> I do not understand the matching logic and how you will avoid false
> positive.
> This last line will e.g. match foo.stderr:exp-ARM.
> And on linux, foo.stderr.exp-x86-* will match foo.stderr.exp-x86-darwin
> 

I think this will be resolved by adopting your suggestion from above,
i.e. using '.' as separators. I was planning to do that.

> 
>> There is still a possibility for false positives in the case where
>> there are several exp files that differ only in the REASON code. To
>> handle that properly we would have to formalize the REASON code.
>> I have no plans to do that as this happens rarely.
> I am not sure to understand the concept of false positive you are
> looking at
> (mentionned here, but also at the begin of the proposal).
> 

...snip...

> I do not understand this second "false positive" concept:
> Imagine that I have a test which (for bad reasons, but let's assume not
> easy to fix)
> can give randomly two outputs:
>   foo.stderr.exp-x86-linux-kfail-notfixablebehaviour1
>   foo.stderr.exp-x86-linux-kfail-notfixablebehaviour2
> The above only differs in the REASON code. Why/how can this give a false
> positive ?

A test that gives random outputs (i.e. behaves non-deterministically)
should not be in the test suite to begin with.
Now suppose the output is deterministic and you're running the tests on
two x86-linux based platforms P1 and P2.
On P1 the output is foo.stderr.exp-x86-linux-kfail-notfixablebehaviour1.
On P2 the output is foo.stderr.exp-x86-linux-kfail-notfixablebehaviour2.

We cannot express that pairing today and the proposal provides no
machinery to do that. All we can do is to compare the stderr.out file
against all provided stderr.exp files (here ...-notfixablebehaviour1 and
...-notfixablebehaviour2).  Should stderr.out from running on P1 match
foo.stderr.exp-x86-linux-kfail-notfixablebehaviour2 we would report that
as a passing testcase. But it isn't. Hence a false positive.

  Florian