Re: [Firebird-test] BBT: Automated QA and TCS

[Pavel C. <pc...@us...>]

Hi,

On 3 Jul 2002 at 10:09, John Bellardo wrote:

> One of the things that would help here is the ability to take generic 
> sql scripts and run them through all the different interfaces (esql, 
> dsql, etc).  This would allow us to drive common aspects of our 
> different interfaces from the same test script.  Write once and test 
> everywhere.  Obviously it will only work for the common subset of 
> interface functionality (dsql doesn't have blob access, for example).  
> But a lot of the tests in TCS are glorified SQL scripts already and 
> would benefit from this.

This would be really nice. But pardon my ignorance (I'm not an ESQL 
guru), how we can run arbitrary SQL scripts via ESQL ? I don't know that 
this is possible directly, so would need some transformation tool (jet 
another kind of GPRE) ?
 
> At one point I had ported most, if not all, of TCS to dejagnu.  So I 
> guess that is also a consideration.  But the interface on that is 
> admittedly poor compared to some of the GUI-based test harnesses.

My main objection to dejagnu/expect solution is that it IMHO really 
doesn't solve the main problem - How tests are written. Actually, TCS 
it's not so bad once you make it running, but tests written in mixture of 
shell script, arcane TCS commands, environment variables (both OS and 
TCS), C/C++, ESQL etc. with use of various external tools (thanks god all 
standard, free or part of TCS suite) and all packed in single text file 
(sorry, I mean BLOB in test database ,-) is really a nightmare. Of 
course, dejagnu/expect can shallow it a bit but at expense that we have 
to rewrite all tests. I we have to write all tests from scratch, I'd like 
try to find more "test writer friendly" solution first. But dejagnu is 
still in play as backup solution.
 
> I think the barrier to entry for the test suite can be higher than that 
> of the engine.  For example, I wouldn't have a problem with the would be 
> tester had to install one or two additional software packages to test 
> (5-10 would be excessive) as long as the process the clearly documented, 
> uses only free tools (of course), and only takes a few minutes aside 
> from download time.  Our testers should not have to learn any additional 
> programming language just to run the tests.  It is a reasonable 
> expectation that they need to know some language to write tests.

I agree, but easy to setup & operable test harness would be a plus. 
Actually, it's not supposed that test harness should be used by more than 
few people -> "platform keepers/packagers and QA specialists" (of course, 
it would be nice if anyone would be able to verify our results, but it's 
not required). But we do not have enough good tests (and we speak about 
thousands tests here) and we need them quickly. That is the main problem 
we face here, regardless of concrete test harness we close with. Since 
Firebird project inception we learned that skilled C/C++ developers are 
scarce to be wasted on something like test cases, even if we'll convince 
them to write them. But there are many Delphi, Java, PHP and SQL 
developers flying around Firebird project that are not C/C++ developers 
and thus disqualified for core engine development, waiting for an 
opportunity to do something useful. We should come with a solution that 
allow these developers to write some tests in their spare time. Not much 
work, not much learning, not much intellectual load. It's IMO the only 
way we can get tests we need and "in time". For comparison, if I'd be 
paid to work on test cases full time, and I'll write (and verify) ten 
tests a day, it will take almost two months for me just to replace TCS 
tests. And we'll need ten times more tests just to match Borland's 
"certification". We need an approach that either allow me to write more 
tests a day (not feasible, new test must be _designed_, written and 
tested and it's a lot of handwork on itself), or allow us to turn in more 
people (usually new, not already involved in Firebird developments). I 
can't see any other way than lowering the entry barrier for test writers. 
But how ? Well, declarative tests can work for some tested areas (common 
SQL, PSQL) but not for all (events, ISQL etc.). But how "declarative 
test" should be "declared" ? How other test should be build ? That is the 
question, dear Horatio :) I've some ideas, but I'd greatly appreciate any 
suggestion about how to make test writing more sexy for anyone.

Best regards
Pavel Cisar
http://www.ibphoenix.com
For all your upto date Firebird and
InterBase information