Hi all,
Although it's not very voiced in public forums, the Firebird project take
quality very seriously - always. We discussed and utilized many
techniques to ensure highest possible quality, and we always look for
improvements in our neverending quest for stability, reliability and
speed of Firebird engine. The very important part in the QA puzzle is the
comprehensive suite of regression tests, that allow us to safely
introduce new features and improvements with minimal or no chance to
break working functionality.
Firebird inherited a test harness called TCS with ~450 tests used by
Borland. The test suite is only a fragment of tests used by Borland IB QA
group and cover only several areas of the product. TCS is also hard to
operate and writing new tests require a lot of knowledge and skills
comparable to skills needed for core engine developers. Due to that
constraints we began to look for TCS replacement with smaller entry
barrier for test developers. Many alternatives have been tested but none
of them attracted new test developers and thus don't took off due to lack
of development resources. Building comprehensive test suite is tedious,
time consuming task that do not fit well to unrestrained open source
development.
IBPhoenix is a company that was always very supportive to Firebird
Project, and have resources to perform development tasks that are needed
but not very popular by volunteers. For last several months, IBPhoenix
sponsored internal R&D to provide a new foundation for a comprehensive
test suite for Firebird. The basic requirements for new test harness were
defined as follows (in order of importance):
- cross-platform
- based on open source (or freely) available tools
- flexibility and extensibility
- simple, text based test format suitable for management in CVS
- low entry barrier for test development
- possibility to make declarative tests along with procedural tests
After evaluation of various options, we decide to use QMTest from
CodeSourcery (www.codesourcery.com). QMTest is open source test harness
written in Python (with CLI and web GUI interface), and fulfil all
requirements described above. Although it's usable "as is" for our
purposes, we extend it with new test and resource classes that make it
Firebird-aware and simplify the development of tests.
Along with development on QMTest, we started a public effort to form a
group of test "designers". For test design you don't really need any
special skills or any software installed beyond Firebird itself, you need
just some knowledge of SQL and Firebird and willingness to spare some
hours on test design. The purpose of test design is to provide a recipe
for test implementation (definition of prerequisites, test steps and
result verification) that would free test developers from most time
consuming part and allow them to focus just on actual implementation.
With many test designers but few test developers we can build a lot of
new tests very quickly. Basic rules for test design are available at
Firebird web in Firebird QA section of Developer's Corner, and you're
welcome to join as test designer. You can also learn QMTest and become a
test developer, or learn Python and become advanced test or QMTest
developer :)
The first goal is not to replace all TCS tests (they work just fine), but
provide complete test coverage for all SQL commands implemented by
Firebird in all their variants. Next step is make tests for all bugs in
our bug tracker.
Although anyone can install and run our test suite, it's supposed that
only core developers, port maintainers and packagers would do that
regularly.
Test design and development is performed as separate sub-project
(Firebird QA) so please, direct your
questions related to our QA efforts to Firebird-test mailing list.
Current status:
The first batch of tests (120) along with QMTest extensions has been
committed to our CVS as QMDB
module, and more (~100) tests were designed and ready for implementation.
The QMDB module is - In
terms of QMTest - a complete, ready to use "test database". Tests are
organized in "suites" based on
(sub)directories. To run them, you'll need QMTest from
www.codesourcery.com, Python (at least 2.1, but
preferably 2.2) and kinterbasdb (open source Python IB/FB connectivity
package from http://sourceforge.net/projects/kinterbasdb), and - of
course - Firebird.
Due to a pending problem with extension classes (running external
programs with stdin/stdout redirection on POSIX platforms) it currently
works only on Windows, but we're working on fix (but you're welcome to
help).
Best regards
Pavel Cisar
http://www.ibphoenix.com
For all your upto date Firebird and
InterBase information
|
