Re: [Firebird-test] On Testing

[Pavel C. <pc...@us...>]

Mark,

On 5 Jul 2002 at 9:31, Mark O'Donohue wrote:

> I'd only consider looking at a new tool, once someone was convinced 
> enough and dedicated enough to actually port the existing tests to 
> whatever new system.  

Actually, I'm that person :) But as I explained in an answer to John 
Bellardo, it's only a starter and not the most important, as current TCS 
tests do not cover the most important areas. The problem is not ESQL, 
TCS, dejagnu or whatever, as I'm willing to learn anything to write tests 
(I had to start learning Python just to play with QMTest customization), 
but the fact that test writing is not a task for single man, because we 
need far more tests than +-400. We need at least ten times more to match 
the Borland's "certification" process. And this is the target I'm working 
on - build a system that can be compared to Borland's QA and exceed it. 
Anything less is IMHO near to waste of resources.

> Following that I'd say take the SQL grammar in parse.y walk through all 
> the tests, perhaps interacting with ib-support / ibdi groups for 
> examples and make sure we have coverage of all SQL statements.

Exactly, that is where I'd like to start !!! Actually, we don't need to 
begin with adaptation of TCS tests - they work fine in TCS once you 
managed to run them, but we need fresh new tests and SQL compliance tests 
are best candidate. What's the REAL problem with new tests is HOW to 
design them, and WHAT environment provide for test writers to make their 
work easy (so more people can write tests) and productive (more test 
would be written in next few months). Both questions are connected, but 
HOW is more important and not related to tools we use, but to DESIGN of 
tests. For example: How we can test CREATE TABLE statement ? Well, 
statement can pass ok, but how we can verify that table was created 
correctly ? By reading system tables ? By populating it with data and  
verification of those data ? By other way ? What are dependencies among 
tests ? For example, we should test the SELECT from system tables first 
to trust results of CREATE TABLE test based on system tables scan. 

Actually, writing test could be task for few, because the hard and most 
time consuming part is the correct test design. We should provide some 
guidelines for test _designers_ how to design tests and how to write this 
design down for test writers. These quidelines can come only from engine 
developers, because they know the dependencies in the code, but then, 
_anyone_ can design new tests that others then implement. If anyone has 
any idea about test design (general or for particular test area), don't 
hesitate to share it !

> (I think doxygen blew a stack on the original fb code, but it and fb2 
> have evolved a little - and Im happy to try again since it'd be useful).

I've noticed only one doxygen short-coming - it's too object oriented 
(aah, so nice class diagrams :-), so the generated doc is not very well-
arranged for code that have a billion of functions and only few classes. 
But maybe I've just overlooked some magic option switch :-)

Best regards
Pavel Cisar
http://www.ibphoenix.com
For all your upto date Firebird and
InterBase information