Re: [Firebird-test] BBT: Automated QA and TCS

[John B. <bel...@cs...>]

Hi,

On Wednesday, July 3, 2002, at 06:31 AM, Paul Reeves wrote:

> Pavel Cisar wrote:
>
>
>> Tests mostly use ESQL for access to the database, instead of DSQL 
>> which is prevailing approach today.
>
> How would you see us testing DSQL? I'd trust the results of an ESQL 
> test over a DSQL test anyday. Writing good (I mean correct) DSQL tests 
> is not easy. At least gpre knows what it is doing.

One of the things that would help here is the ability to take generic 
sql scripts and run them through all the different interfaces (esql, 
dsql, etc).  This would allow us to drive common aspects of our 
different interfaces from the same test script.  Write once and test 
everywhere.  Obviously it will only work for the common subset of 
interface functionality (dsql doesn't have blob access, for example).  
But a lot of the tests in TCS are glorified SQL scripts already and 
would benefit from this.

>
> [...]
>
>> We'll need more friendly TCS. Open source QMTest 
>> (www.codesourcery.com) is an option, but you can suggest other tools 
>> (including home-grown) as well.
>
>
> Have you seen QAT yet? (http://qat.sourceforge.com). It is another OS 
> test harness, this time written in Java. I haven't played with it 
> seriously but the fact that it is in Java makes it more interesting to 
> me.
>

At one point I had ported most, if not all, of TCS to dejagnu.  So I 
guess that is also a consideration.  But the interface on that is 
admittedly poor compared to some of the GUI-based test harnesses.

> My main objection (in fact my only objection) to QMTest is that it 
> requires Python. It is another thing to install and configure and it 
> quite probably requires learning a bit of python, too. That said, I'm 
> not stuck on this. A harness that is easily configurable and allows 
> easy extension addition to the test 'database' is the most important 
> thing. We need to reduce the barriers to test writing as much as 
> possible.

I think the barrier to entry for the test suite can be higher than that 
of the engine.  For example, I wouldn't have a problem with the would be 
tester had to install one or two additional software packages to test 
(5-10 would be excessive) as long as the process the clearly documented, 
uses only free tools (of course), and only takes a few minutes aside 
from download time.  Our testers should not have to learn any additional 
programming language just to run the tests.  It is a reasonable 
expectation that they need to know some language to write tests.

-John