Re: [Firebird-test] On Testing

[Mark O'D. <mar...@lu...>]

Hi Pavel

Pavel Cisar wrote:

>  
>
>Exactly, that is where I'd like to start !!! Actually, we don't need to 
>begin with adaptation of TCS tests - they work fine in TCS once you 
>managed to run them, but we need fresh new tests and SQL compliance tests 
>are best candidate. What's the REAL problem with new tests is HOW to 
>design them, and WHAT environment provide for test writers to make their 
>work easy (so more people can write tests) and productive (more test 
>would be written in next few months).
>
The real problem with test cases, is not writing them, its handling the 
output.  The crucial areas being:

1) parsing output data.
The system needs to be robust enough to handle changes in 
environment/format (for example Claudio's better error handling broke 
80% of the TCS test, since the output changed slightly).  So look 
carefully at the output parsing mechanisms,  The dejagnu/expect one is 
fairly good, you pattern match on what you want it ignores the rest. TCS 
does a simple diff on it's output, but more flexible one is desirable - 
for example in expect to test first/skip clauses, I just counted number 
of records returned, rather than match all output.

2) reviewing differences
When output differs from prior run, a good comparison method is needed 
(diff is good) but tcs comparison is poor (mainly since it splits stdio 
and stderror).  Flat files for data & results also help versioning.

Im not real worried about a gui, I just want to do:

 $make test.
and a diff for output

Much like junit (which I rather like), although we're not really in a 
position to use it.

Test design and init scripts and test groups etc, are not really big 
issues, when you get into it.



> Both questions are connected, but 
>HOW is more important and not related to tools we use, but to DESIGN of 
>tests. For example: How we can test CREATE TABLE statement ? Well, 
>statement can pass ok, but how we can verify that table was created 
>
The How is as follows, your test should involve script to check the 
results.  In expect something like:

expect_results = "TEST                  NUMERIC(18,0) Not Null";

send "create table fred (test integer);"
expect $prompt
send "show table fred;"

result = ""
expect {
       "$prompt":  break;
      "^*\n"  :     result = $result + currentLine
      -timeout :
}

if $result <> $expect_result) then
    fail_test "unexpected results"
fi

Other scripting things shouldbe similar, (Course if your smart you'll 
put the results into a file, and paramaterise the input).  How you 
organise the into groups/modules etc is then up to you.   It's simple 
but effective and very flexible.  

Even if you have a gui, loadrunner, QAtest and all those guys all end up 
with a script interface.

Btw: you should check for a python interface to dejagnu, it wouldn't 
supprise me if there was something there.

Im not religious about the tcl/expect but I didn't see anything better 
that covered the basics, and some of those other options cost lots of 
money - but if you find something I willing to listen.

Cheers

Mark

>  
>