[Firebird-test] Firebird Testing - A plan for discussion

[Leyne, S. <sl...@at...>]

All,

After discussion amongst the Firebird Admin Team, I've taken on the role
as the Testing Project Co-coordinator (actually, Mark O'Donohue gave me
an incentive).

Of all aspects of the Firebird, testing has garnered the least attention
so far and is the least understood by most of the members.  With the
upcoming v1.0 release, this needs to change.

So, please find below my proposal (open for discussion) of the
plan/priorities for the long-term testing aspects of the Firebird
project.

Realistically, however, the main focus of the testing effort for
Firebird 1.0 will be on testing the database engine.  The structure of
the peripherals and even most of the engine still (despite bug fixes and
the occasional feature) does not differ remarkably from the original
code, so most of the testing done by Borland is still applicable to the
Firebird codebase.  As time goes on an Firebird evolves and begins to
differ from the InterBase original we expect the test suites to grow
with the project and eventually cover all the features of Firebird both
new and original.

Finally, an opensource and free Firebird project is only going to
succeed 
if people step up, participate and contribute.  In the current testing 
phase this is an opportunity to contribute for those of us who are to 
directly benefit from the Firebird project.  To that end I/we welcome
all volunteers.


Sean

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - -

Introduction

The current Firebird release policy requires that all releases (whether
final, test, dev, alpha or beta) successfully pass through the complete
TCS Suite as it exists today.

The TCS Suite, was released by Borland, and it covers a broad range of
the engine functionality and was developed of the course of many years.
So in principle all we need to do is compile the engine and run the
tests to check our changes.

But, as was the case with the engine source code, life is not quite that
simple.

The suite (like the source code) has suffered from some disrepair and
was designed to work within a very specific installation framework
within the Borland lab (especially the Win32 environment).  This has
meant that only the most knowledgeable can effectively build and run the
test suite.

A lot of the tests relate to aspects of the project that are no longer
heavily used, gdml for instance and are lacking in any functionality to
cover the newer ways that databases are used in more detail, the java
interface for instance.

Finally, Borland did not choose to release the stress and volume testing
routines which are used as part of "certification" process.

All, however, is not lost.

First, it is very important to remember that the Firebird project
members are proving every day that we have the technical expertise to
attack any problem (a c++ version of the engine anyone?) and that our
knowledge of the engine is growing.

There are a number of additional test suites we can draw on to
supplement the existing tests.  The PostgreSQL test suite, the NIST3 sql
coverage test suite (which was also used as a major? part of the
internal Borland QA suite), the TPS3 (??) benchmark testing suites and
the Sun jdbc compliance test suite.


1.	Creation of "install and go" Testing Binaries

In order for testing to become widely accepted and pursued, we need to
make the process of running the testing programs as simple as installing
the client/engine binaries themselves.

This extends to developing instructions on: the installation and
execution for the testing programs, reporting test failures to the
project team, building and submitting new scripts to the master project.


2.	Elimination of the database for storing scripts and results

(This is a personal pet peeve of mine -- I've never understood the logic
of need the engine to be running in order to test the engine!).

This will require a change to the testing program logic but I feel that
it is necessary for three reasons.

First, storing the scripts in the database means that changes to scripts
can't be easily tracked within the CVS structure.

Second, storing the scripts and results as simple TXT files (we can
argue about the format choice later) would eliminate the need for the
engine to be installed on the local PC (running the testing programs)
and thus would further reduce the footprint and 'bar' for those looking
to contribute to the testing process.

Finally, there are no tools to enable people to read/edit the scripts
within the database.  This makes the testing even more difficult for
community members to understand and therefore contribute to.


3.	Need to develop engine stress/load testing programs.

This is something which will require the most effort.

First, a clear statement of the goal needs to be developed.  The
information/requirements gathering and discussions are already underway.
All input is appreciated.

Second, an architecture/approach needs to be selected -- the approach
will probably change over time but we need to start somewhere (and
besides some of us have nothing but free time to re-architect the
programs <huge grin>).

It has been suggested, and agreed by a number of people, that if the new
stress/load testing routines are designed correctly, then it could also
form the basis of a test bed for a more general test suite, replacing
the current TCS program.

Finally, the programs need to be developed.


4.	Need to develop complete client data access testing programs.

There are no tests for any of the client data access tools, so any new
development which takes place on ODBC/ADO/JDBC, needs to go through a
testing regiment of it's own. 

This task should be the responsibility of each development team and be a
required part of their development plans -- so much so, that sucessfull
testing should be required prior to a release being made (ODBC/JBDC
projects are you listening???).


5.	Documentation of Testing tasks

This topic is a little wide ranging.

It starts with an update on the status of the testing programs for the
various platforms.  For example, can the tests be run against a remote
engine or only local engines, what is the status of the Win32 testing
install/environment (as far as I know it needs a lot of work). 

From there, we need to document and categorize all of the exiting
scripts, test groups and expected results.  

From there, we need to add new tests which attacks the bugs which have
been fixed in the engine as well as those bugs which have been found and
remain unresolved (this may require adding logic to handle expected test
failures).  [This could produce interesting results when run against the
Borland "certified" engine]