[Firebird-test] Testing program

[Leyne, S. <sl...@at...>]

All,

I think we need to step back from the C++ / Delphi/Kylix debate (one
which will have no end) and review the requirements for the testing
solution.  (This discussion also really belongs in the testing list)

The basic requirements for the engine/API testing programs should be:

1)	The test scripts and the expected results need to be stored
outside of any database in a text (or xml) format.  The file format
should include a header which explains the nature of the test.  (You
would no longer need to have FB installed in order to test FB!)
 
2)	The testing scripts must support the ability to define groups of
tests (another script file in the scripts subdirectory).  The group
script would contain special tags pointing to the scripts to be
executed.  That way the NIGHTLY.GRP file would be a group test script
which could be run against the nightly builds.

3)	The actual testing program(s) should be able to run from any
client platform against any server platform.

4)	Wherever possible, binaries of all testing programs should be
provided to make the task of running the tests as easy as possible for
all testers. 

5)	Although the testing scripts would become part of the CVS tree,
the scripts and expected results need to be available as a separate ZIP
or TAR download.  This combined with the testing binaries will allow for
anybody to download the program binary of their choice and the scripts
and be testing within minutes.

6)	All testing programs need to be able to produce identical
(byte-level duplicated) results - i.e. a C++ program should produce the
exact same result file as a Delphi/Kylix program.  So, if your "visual
inclined" (like me) and hate command line utilities, you can run the
Windows GUI version of the testing program and still be able to know
that the test were successful.

Other requirements will apply to the other testing programs which will
be required for ODBC, JDBC, the new Java Client etc.  but this should be
the start of something we all agree on.


Sean