[Firebird-test] Advancements in Firebird QA

[Pavel C. <pc...@us...>]

Hi all,

Although it's not very voiced in public forums, the Firebird project take 
quality very seriously - always. We discussed and utilized many 
techniques to ensure highest possible quality, and we always look for 
improvements in our neverending quest for stability, reliability and 
speed of Firebird engine. The very important part in the QA puzzle is the 
comprehensive suite of regression tests, that allow us to safely 
introduce new features and improvements with minimal or no chance to 
break working functionality.

Firebird inherited a test harness called TCS with ~450 tests used by 
Borland. The test suite is only a fragment of tests used by Borland IB QA 
group and cover only several areas of the product. TCS is also hard to 
operate and writing new tests require a lot of knowledge and skills 
comparable to skills needed for core engine developers. Due to that 
constraints we began to look for TCS replacement with smaller entry 
barrier for test developers. Many alternatives have been tested but none 
of them attracted new test developers and thus don't took off due to lack 
of development resources. Building comprehensive test suite is tedious, 
time consuming task that do not fit well to unrestrained open source 
development.

IBPhoenix is a company that was always very supportive to Firebird 
Project, and have resources to perform development tasks that are needed 
but not very popular by volunteers. For last several months, IBPhoenix 
sponsored internal R&D to provide a new foundation for a comprehensive 
test suite for Firebird. The basic requirements for new test harness were 
defined as follows (in order of importance):

- cross-platform
- based on open source (or freely) available tools
- flexibility and extensibility
- simple, text based test format suitable for management in CVS
- low entry barrier for test development
- possibility to make declarative tests along with procedural tests

After evaluation of various options, we decide to use QMTest from 
CodeSourcery (www.codesourcery.com). QMTest is open source test harness 
written in Python (with CLI and web GUI interface), and fulfil all 
requirements described above. Although it's usable "as is" for our 
purposes, we extend it with new test and resource classes that make it 
Firebird-aware and simplify the development of tests.

Along with development on QMTest, we started a public effort to form a 
group of test "designers". For test design you don't really need any 
special skills or any software installed beyond Firebird itself, you need 
just some knowledge of SQL and Firebird and willingness to spare some 
hours on test design. The purpose of test design is to provide a recipe 
for test implementation (definition of prerequisites,  test steps and 
result verification) that would free test developers from most time 
consuming part and allow them to focus just on actual implementation. 
With many test designers but few test developers we can build a lot of 
new tests very quickly. Basic rules for test design are available at 
Firebird web in Firebird QA section of Developer's Corner, and you're 
welcome to join as test designer. You can also learn QMTest and become a 
test developer, or learn Python and become advanced test or QMTest 
developer :)

The first goal is not to replace all TCS tests (they work just fine), but 
provide complete test coverage for all SQL commands implemented by 
Firebird in all their variants. Next step is make tests for all bugs in 
our bug tracker. 

Although anyone can install and run our test suite, it's supposed that 
only core developers, port maintainers and packagers would do that 
regularly.

Test design and development is performed as separate sub-project 
(Firebird QA) so please, direct your 
questions related to our QA efforts to Firebird-test mailing list. 

Current status:

The first batch of tests (120) along with QMTest extensions has been 
committed to our CVS as QMDB 
module, and more (~100) tests were designed and ready for implementation. 
The QMDB module is - In 
terms of QMTest - a complete, ready to use "test database". Tests are 
organized in "suites" based on 
(sub)directories. To run them, you'll need QMTest from 
www.codesourcery.com, Python (at least 2.1, but 
preferably 2.2) and kinterbasdb (open source Python IB/FB connectivity 
package from http://sourceforge.net/projects/kinterbasdb), and - of 
course - Firebird. 
Due to a pending problem with extension classes (running external 
programs with stdin/stdout redirection on POSIX platforms) it currently 
works only on Windows, but we're working on fix (but you're welcome to 
help).


Best regards
Pavel Cisar
http://www.ibphoenix.com
For all your upto date Firebird and
InterBase information