Re: [Firebird-test] Testing Firbird --- SI-T1.0.0.366 Firebird Be

[John B. <bel...@cs...>]

Hi,

On Sunday, August 26, 2001, at 08:44 PM, Mark O'Donohue wrote:

>
> Hi Sean
>
> Leyne, Sean wrote:
>
>>
>> What would it take to have the reports produce a more 'intelligible'
>> output?
>>
>> This output seems absolutely useless!
>>
> It is, what John is working on will enable us to do a unified diff (aka 
> the code changes), and to mix the stdout/stderror streams so the output 
> appears normal and in context.
> It's not far off, last report was there were only 8 TCS tests that when 
> converted still produced different results to the TCS run - but I 
> should let him describe that.

I have just committed (as anyone on firebird-checkins can attest to :-) 
a whole new test suite.  Most of it consisted of the flatfile data 
extracted from TCS, but there was an updated test script in there to.  
Right now my latest build passed all but 2 or 3 tests using the new test 
suite.  Here is a [maybe not so] brief description of the suite:

It is a straight lift of TCS.  I have not modified any of the tests or 
the test data itself.  I'm running of the same blob's used in TCS, 
except they are in flat files now.  What the test script does is emulate 
(to the best of my TCS knowledge) all aspects of the TCS pattern 
replacement, test script construction, and data file construction 
process.

Once the test data has been extracted and the environment configured 
properly the test is run.  The output of the test is captured in the 
same order a user would see it (stdout/stderr intermixed).  The current 
TCS expected output is in the format stderr first, then stdout.  This is 
one of the things that makes determining where a test failed difficult.  
As the new tcs.exp test reads the lines (one at a time) it matches them 
against the original TCS expected output to check for failure.  A fail 
(expected or unexpected) is reported.  The code also handles expected 
output files with intermixed stdout/stderr.  This format is what Mark 
was talking about in the quoted message.  After we are satisfied with 
the correctness of tcs.exp we can use it to generate new intermixed 
expected output files that will be easier to understand.

Current tcs.exp does not contain any support for creating new tests (ie, 
capturing the output).  That support needs to be added, but it won't be 
very difficult.

tcs.exp also lacks the correct hooks into the build/install 
environment.  That means it only works on my machine :-)  But the good 
news is all it takes to get working on another machine is correctly 
setting a couple of environment variables.  Tighter integration with the 
build process (something obtainable for FB2, but probably not worth the 
effort for FB1) can allow full tcs.exp testing against a built engine 
without installing it and while another engine is still running on the 
same host!

tcs.exp is written in Tcl using expect and dejagnu.  These programs are 
all available for win32 (Mark had an earlier version of tcs.exp running 
in win32 under vmware) and available from cygwin.  I'm not sure how 
feasible installing pure win32 ports the the programs is.  It is quite 
possible the tcs tests assume the presence of a program (for example sh) 
that tcs.exp doesn't explicitly rely on.

tcs.exp still requires a compiler to be present to test.  But it also 
has the notion of sub series.  Using sub series we could define a set of 
tests that doesn't rely on the compiler (for example the isql tests).  
These tests could be run on an end-users machine assuming they has 
cygwin installed (for the afore mentioned dependencies).  They wouldn't 
need the MS (or whomever) compiler.

Well, I think I've written enough for the time being...

-John