Re: [Firebird-test] QMTest runs against fb1.5

[Pavel C. <pc...@us...>]

Mark,

On 23 Nov 2003 at 19:25, Mark O'Donohue wrote:

> The first run I get:
> 
> Error: 	314

Those are for sure bug tests (bugs.qms test-suite). Those are not yet 
implemented, just a stubs with problem/bug description. Do not run those, 
only isql.qms test-suite is ready right now.

> Fail: 	28
> Pass: 	199
> 
> Total: 	542
> 
> It's a first run, and most of the errors, will not be errors as such,
> but differences from expected outcomes.
> 
> So a few questions.
> 
> 1) In QMTest can it store multiple configurations for results?.
> I obviously would like to create a fb1.5-linux expected output database,
> but don't want to muck up your current ones?

Of course, you can save expectation/outcomes under any name and load it 
later.

> 2) When I review a test run, and verify that the 'new' results are ok,
> how do I accept them so they replace the current expected output?

Not by one click, but quite simple. Copy *stripped* stdout (or stderr) 
from outcome details of failed test (best done in gui mode), enter the 
edit mode for test and replace the stdout (stderr) text with new one. Re-
run the test for verification. 

NOTE: Stripped output is the output with internally defined "ignore 
patterns" filtered out (i.e. isql prompts etc.)

Some more notes:

1) If the difference is only in particular FB version (i.e. 1.5, but 
correct in 1.0), then we should duplicate that test as new one in 1.5 
specific test-suite. There are no such version specific test-suite, so we 
have to create one for 1.0 and 1.5 (new1.0 and new15 suites have 
different purpose!) and move the old test to spec10.qms and new one into 
spec15.qms.

2) If the difference come from isql or configuration (file path etc.), 
it's better to define "replace" regular expression for test with empty 
replacement (stands for ignore). For more "common"
 differences it's better to filter them out in extension class (like isql 
prompts are removed).

Best regards
Pavel Cisar (ICQ: 89017288)
http://www.ibphoenix.com
For all your upto date Firebird and
InterBase information