Re: [Firebird-test] Is this what you are looking for

[Pavel C. <pc...@us...>]

Mark & all,

On 3 Aug 2002 at 16:36, Mark O'Donohue wrote:

> First it's important to identify what you want to test.
>
> Perhaps as a starter get the sql stmts from parse.y and break them into 
> stmts (or probably clauses).
>
> This will also make it easier to distribute the work amongst different 
> people without much duplication or overlap.
> 

You are right, but it's a helluva lot of work ! Making such a list will 
take some time, so we need to improvise a bit in meanwhile. I'm working 
on how-to's and to-do lists for web publishing (due date at the end of 
next week). That doesn't mean that work should stop until I finish it :) 
I'll answer all questions in this list, and any comments are more than 
welcome. I'm very happy about the interest the QA gets, but it's a work 
in progress, so forgive some chaotic steps please :)

Anyway, I'd like build tests in dependency order rather than statement 
order. For example, tests for ref. constraints would need table creation, 
table inserts and table scans for initialization and failure checks, so 
we need these tests first to assure validity of dependent tests.

I'm working on basic prerequisite tests to assure validity of empty 
database (data in system tables). That's connected to basic SELECT tests 
for selects from single table - for all or listed fields. All other areas 
are now open for all, but most needed tests right now are tests for 
creation of domains and tables (additional checks should use simple 
selects from system tables), for start without more complex clauses like 
PK, indexes etc. just various data types, row lengths etc.

That doesn't mean that tests for other areas are not welcome! We greatly 
appreciate any tests, but tests that are at the bottom of dependency 
graph would be implemented first. And after all, requirements for all 
tests would help us to map the dependencies (SQL space is huge :), so 
don't hesitate to design tests for area you know best.

> Then for each stmt, lets take FIRST/SKIP you need to identify all the 
> features of that stmt you want to test.
> 
> 1. range at start
> 2. range at end
> 3. range with subselect
> 4. range with no/incomplete data
> 5. illegal ranges
> 6 .range with sort
> etc.
> 
> The you design the tests to cover the above features, say 4. no 
> incomplete data
> 
> 4a select skip 10 first 5  - with zero data.
> 4b                                      - with 2 rows
> 4c                                     - with 7 rows
> 4c                                     - with 1 row
> 4c                                      - with 10 rows
> etc...
> 
> 
> That will give a bit more structure to you test process.  Focussing on 
> testing all the legal paths first - does it work correctly as specified.

Yes, that's the right way and exactly follow the prime directive - simple 
test cases, one aspect at a time. And positive tests first! Of course, 
some features like ref. constraints are better checked in negative tests.

Other big group of tests we need are test cases for each bug database 
entry. But these tests would be more complex, so we should defer them for 
now (with exception for really simple cases).

> To run this in a practical way you are going to need a list of who is 
> working on what stmt's or clauses, and you will need to assign and 
> identifiers to test features, test cases, etc.

Until to-do list would be ready, all test designers should claim their 
area of work in this list. I'll incorporate them in to-do list on the 
fly.

Best regards
Pavel Cisar
http://www.ibphoenix.com
For all your upto date Firebird and
InterBase information