Re: [Firebird-test] How to manage testing for FB1.5

[Pavel C. <pc...@us...>]

Hi,

On 18 Dec 2002 at 23:19, Ignacio J. Ortega wrote:

> I've playing with great success with qmTest suite, and it's is really
> promising, a big step forward, thanks for the effort..

Nice to hear that :)
 
> I'm struggling testing FB 1.5, and i've found a number of failures
> related to the number of pages a empty databse has.. it seems fb1,5 has
> a little bigger footprint in pages ( maybe related to the new indexes in
> metatadata tables), so it seems this tests has been developed against
> fb1.0..

Yes, it was developed around 1.0, as I need to use stable engine as 
reference platform.

> my question is,  how it's supposussed the test database will be
> organized to test different fb versions?, or maybe we only consider the
> development branch under QA testing? ( so this finding is really a bug
> :)..

Well, that wasn't yet decided. TCS use different db records for platform 
and version specific tests, so TCS will "check-out" tests according to 
configuration on run. QMTest haven't (yet) such capability, so we have to 
decide on "workaround". We can try to use organizational approach or try 
to change QMTest to support platform/version tests/config like TCS. I'd 
like go with test db organization rather than QMtest modification from 
obvious reasons.

Use of CVS branch feature is not very good, because that impose a burden 
to merge them often (both are supposed "stable").

I'd propose the next scenario:

- We can use stable and development trees for tests. Stable would be used 
for current stable FB version (1.0 right now), development for current 
development FB (1.5 right now). 

- Stable tests would be used for maintenance release and as regression 
basis for dev engine.

- Development tree would contain only adapted tests and tests for new 
features. We can run stable and dev tests against dev engine and use 
saved list of expectations for stable tests so test failure would be 
treated as pass if it's expected to fail :) Dev suite should pass.

- When dev engine reach stable release, we'll merge dev to stable and 
clear out dev for another round.

Of course, current directory structure of QMDB have to be changed, but 
we'll still use"natural" test suites - subdirectories. So we should have 
"stable" and "dev" subdirectories under QMDB, and "purpose" 
subdirectories like "create database", "alter database" etc. beneath 
them.

Comments ? Suggestions ?

Best regards
Pavel Cisar
http://www.ibphoenix.com
For all your upto date Firebird and
InterBase information