Re: [Firebird-test] Congratulations on the work for the new tests!

[Pavel C. <pc...@us...>]

Sean,

On 3 Feb 2003 at 20:52, Leyne, Sean wrote:

> Even though I can't run the tests, I must say that I am very impressed.

Thanks, but we're just at the beginning :-)
 
> 1)	I note that more of the tests (I haven't look at all of them)
> don't distinguish between Dialects, is this deliberate?

Yes and No. I've decided to use dialect 3 as a main, implicit dialect, 
and all tests made till today should be for dialect 3. Legacy dialect 1 
should be handled later in separate tests.
 
> I suppose we could create different/separate versions of the tests (i.e.
> "avg_01_D3.qma"), should dialect be an issue. (and assume that all tests
> are Dialect 1 unless noted otherwise)

I would rather put them to separate directory (test suite), but yes, they 
should be based on current tests.
 
> 2)	Following your directory structure, I suppose that we
> could/should start a new directory for the SF Bug tracker items --
> SFTracker.qms.  

The current directory/suite structure is temporary. I'm sure we will 
rearrange them to better one as test database grows and we'll finally see 
what all different groups of tests we have.

Because we found some bugs when developing core SQL reference tests in 
isql suite, I'd like add the SFBug code as a standard part of test 
description. 

> Each test could the use the SF # as part of the filename --
> SF232023.qma and SF232023_1.qma (part 1 of a series) 

Yes, I intended to use this naming convention for tests in Bugs suite 
intended for test cases developed exclusively for bugs from SFBug entries 
to check their fix, but other tests can also link to bug (the chicken and 
egg problem :-)

BTW, there is also a problem with testing of individual Firebird 
versions/build. This should be solved by saved "expectation" files 
(stored in CVS too), because some tests should fail (known bug) while 
others not. test outcome expectation files should filter out expected 
failures, so only "real" ones shows up.

Please note that some tests simply doesn't work (two tests AFAIK) due to 
pending problem with use of accented characters in test definition. It's 
pure QMTest XML test storage problem.

Best regards
Pavel Cisar
http://www.ibphoenix.com
For all your upto date Firebird and
InterBase information