[Firebird-test] Invitation to disscussion about Firebird QA

[Pavel C. <pc...@us...>]

Hi all,

Because Firebird Project is about to completely switch all development to 
new source tree (called Firebird 2) converted from C to C++, and large 
changes in engine subsystems are planned or already implemented (like new 
memory management, use of exceptions, database aliases etc.), the 
effective QA is more important than ever.

You may know that along with IB, Borland open-sourced a test control 
system (with few hundred tests) for it called TCS (available as a 
separate project on SF or in Firebird's cvs repository). This test 
harness is able to run on multiple platforms (is written in C) and is 
used by Firebird Project to test every official Firebird release. But the 
total amount of tests is small (Borland released only a small subset of 
all available tests), and because tests are mostly written in C and ESQL, 
and with extensive use of tools like QLI, ISQL etc., it's very hard for 
arbitrary developer to change or write new tests. One must have skills 
and knowledge at least as any core engine developer, and in many cases 
much deeper.  Many tests are also poorly designed, and the usability of 
test reports is questionable (it's very hard to decipher the exact reason 
of any failure and failures are often false, i.e. test environment 
errors). Under given circumstances, it's impossible to use TCS as a 
framework for Firebird QA in scale we need it. We were aware of that for 
long time, and after months of internal R&D it finally seems that we have 
a solution ready for use.

As a core test framework, we'd like try to use the QMTest - an open 
source test tool written in Python by CodeSourcery (available at 
http://www.codesourcery.com). QMTest has a CLI and Web interface, is able 
to perform distributed testing, tests and test resources are stored in 
XML files suitable for cvs and much more. The most important thing for us 
is that's able to run on any platform where Python is available (AFAIK 
all platforms supported by Firebird) and that test could be written in 
any language. We'd like to use Python with kinterbasdb or gvib for access 
to FB (both are also open source packages) for core SQL tests, but some 
test could (or must) be written in other languages like Java, C/C++, 
Delphi/Kylix etc.

Because we're about to build the whole new suite of tests, we would 
greatly appreciate ANY help. If you're willing to share your expertise 
about automatic testing, Python, Java, C/C++, SQL, or even better, if 
you're willing to DESIGN (Python is not necessary, just SQL) or WRITE 
some tests (mostly in Python, but Python is beautiful and simple 
language, and writing tests is not hard, just boring :), please join us 
in Firebird-test mailing list (subscription details:  
http://www.firebirdsql.org/fbnew/index.php?op=lists#fb-test)

Best regards
Pavel Cisar
There is nothing wrong with InterBase
that Firebird can't fix for you
http://www.firebirdsql.org