Re: [Firebird-test] QMTest runs against fb1.5

[Mark O'D. <mar...@fi...>]

Thanks for that Pavel.

I'll have some more questions in the next day or so, when I try again to 
validate my linux build through your process.

More of the 'Fails' are the bugs,

And from my checking a lot of the errors are correct just different 
output to the fb1.0 results that are stored (for example select of 
avaliable character sets returns different but correct for fb1.5  results).

But more when I try to save outcomes and try and edit them for my 
linux-fb1.5 results.


But while Im here, where are the test output filters configured?


Cheers

Mark


Pavel Cisar wrote:
> Mark,
> 
> On 23 Nov 2003 at 19:25, Mark O'Donohue wrote:
> 
> 
>>The first run I get:
>>
>>Error: 	314
> 
> 
> Those are for sure bug tests (bugs.qms test-suite). Those are not yet 
> implemented, just a stubs with problem/bug description. Do not run those, 
> only isql.qms test-suite is ready right now.
> 
> 
>>Fail: 	28
>>Pass: 	199
>>
>>Total: 	542
>>
>>It's a first run, and most of the errors, will not be errors as such,
>>but differences from expected outcomes.
>>
>>So a few questions.
>>
>>1) In QMTest can it store multiple configurations for results?.
>>I obviously would like to create a fb1.5-linux expected output database,
>>but don't want to muck up your current ones?
> 
> 
> Of course, you can save expectation/outcomes under any name and load it 
> later.
> 
> 
>>2) When I review a test run, and verify that the 'new' results are ok,
>>how do I accept them so they replace the current expected output?
> 
> 
> Not by one click, but quite simple. Copy *stripped* stdout (or stderr) 
> from outcome details of failed test (best done in gui mode), enter the 
> edit mode for test and replace the stdout (stderr) text with new one. Re-
> run the test for verification. 
> 
> NOTE: Stripped output is the output with internally defined "ignore 
> patterns" filtered out (i.e. isql prompts etc.)
> 
> Some more notes:
> 
> 1) If the difference is only in particular FB version (i.e. 1.5, but 
> correct in 1.0), then we should duplicate that test as new one in 1.5 
> specific test-suite. There are no such version specific test-suite, so we 
> have to create one for 1.0 and 1.5 (new1.0 and new15 suites have 
> different purpose!) and move the old test to spec10.qms and new one into 
> spec15.qms.
> 
> 2) If the difference come from isql or configuration (file path etc.), 
> it's better to define "replace" regular expression for test with empty 
> replacement (stands for ignore). For more "common"
>  differences it's better to filter them out in extension class (like isql 
> prompts are removed).
> 
> Best regards
> Pavel Cisar (ICQ: 89017288)
> http://www.ibphoenix.com
> For all your upto date Firebird and
> InterBase information
> 
> 
> 
> 
> -------------------------------------------------------
> This SF.net email is sponsored by: SF.net Giveback Program.
> Does SourceForge.net help you be more productive?  Does it
> help you create better code?  SHARE THE LOVE, and help us help
> YOU!  Click Here: http://sourceforge.net/donate/
> _______________________________________________
> Firebird-test mailing list
> Fir...@li...
> https://lists.sourceforge.net/lists/listinfo/firebird-test
> 
>