RE: [Firebird-test] BBT: Automated QA and TCS

[Leyne, S. <sl...@at...>]

Paul et al,

> From: Paul Reeves [mailto:pr...@ib...]

> Pavel Cisar wrote:
>=20
> > Tests mostly use ESQL for access to the database,=20
> > instead of DSQL which is prevailing approach today.=20
>=20
> How would you see us testing DSQL? I'd trust the results of=20
> an ESQL test=20
> over a DSQL test anyday. Writing good (I mean correct) DSQL=20
> tests is not=20
> easy. At least gpre knows what it is doing.
>=20
> This is not a reason to ignore testing DSQL, especially as gpre=20
> generates undocumented api calls in some cases. But having played a=20
> little with writing gpre scripts I think the learning curve is more=20
> shallow. Test writers can concentrate on writing code that will=20
> accomplish a test. Writing DSQL will turn into a test of the=20
> programmers=20
> C/C++ abilities - at least in the beginning.
>=20
> I think one of the goals of the QA group is to create an environment=20
> where tests can be created quickly and easily by many=20
> developers. ESQL makes this a lot easier.


Just as a reminder of a historical posting, Reed Mideke wrote:

> > The current ISQL approach virtually eliminated these type of errors,
> > saving testers from hunting down these 'false positive' results.
> > Furthermore, when it comes to diagnosing anomolies in data=20
> results, ISQL
> > has been used as the "gold standard" for determining=20
> whether a problem
> > is engine or data access layer related.
> >=20
> Emm, actually, embedded GDML (i.e. GPRE) is the gold standard.
> isql only lets you test DSQL.=20

Accordingly, perhaps we need to balance the use of GPRE based tests and
DSQL/isql based test.

Ideally, the GPRE test should cover the very basic elements of the
database/engine and those features which can't be tested any other way.
Once the basics have been tested then other tools (like isql) can be
used to perform the reminder (majority) of the tests.  In the long run,
these basic tests should change less frequently than the need to create
new test to address DSQL issues.

To address the deployment issues, installing/configuring a C/C++
compiler, we could build binary packages which could be installed by
users/testers (this is in addition to the test package -- no CVS access
should be required).  Thus, only 1 person would need to compile and
build the binary package, and another to build/post the package of
testing scripts.=20

Then, the thousands of users can contribute to the overall testing
process.

...

> My main objection (in fact my only objection) to QMTest is=20
> that it requires Python. It is another thing to install and
> configure and it quite probably requires learning a bit of
> python, too. That said, I'm not stuck on this.

I also share this opinion.  But at the same time we need a solution
which:

1) runs well on Windows (I lot of our users and potential new
developers/testers use Windows)

2) runs on all platforms without requiring too many supporting tools

3) provides a good/meaningful display of the test results and error
details

4) stores test scripts in individual text/xml files

5) can be installed with a straight-forward set of installation notes
that even I can follow (in other words, idiot-proof!).

It seems that QMTest meets these requirements, but I'm open to others.


Sean