firebird-test — Firebird engine testing, test flight/suite discussion/information

[Firebird-test] select MAX() not used INDEXES to run

[Carlo P. <ca...@ug...>]

Hi,

I'm testing firebird with a table with 65000 records. When I try to:

select MAX(COD_VALUE) from TABLE_TEST;

The plan used by firebird is NATURAL. This is strange to me because I have a
index for field COD_VALUE. I had to implement a stored procedure + a
descending index for field COD_VALUE to get, more quickly, the max value for
COD_VALUE.

Know anyone about this? I'm wrong ?

-Carlo

[Firebird-test] test

[Carlo P. <ca...@ug...>]

ca...@ug...ca...@ug...

Re: [Firebird-test] SS running 7 interbase-processes on RedHat??

[Pavel C. <pc...@us...>]

Hi,

This question actually belong to ib-support list, but here you are 
anyway :) But please use ib-support in future, this lists is 
dedicated to Firebird QA efforts.

On 21 Jan 2002, at 23:52, t eriksson wrote:

> I start firebird SS from /etc/init.d/firebird start
> and without starting I can't connect so
> I guess that proves I'm running SS?

Yes.
 
> Still, ps shows ibguard + 3 processes running interbase.
> 
> After a while using interbase,
> there are 7 interbase processes running interbase,
> is this right? I thought SS ran as one single process?

Those are actually threads. Threads are reported by most Linux 
utilities as processes (actually, they are lightweight processes). 
Take a look at other parameters reported by ps command, you'll 
see that these "processes" share memory etc.

Best regards
-- Pavel

There is nothing wrong with InterBase
that Firebird can't fix for you
http://www.firebirdsql.org

[Firebird-test] SS running 7 interbase-processes on RedHat??

[t e. <no...@ho...>]

Hi everyone,

I start firebird SS from /etc/init.d/firebird start
and without starting I can't connect so
I guess that proves I'm running SS?

Still, ps shows ibguard + 3 processes running interbase.

After a while using interbase,
there are 7 interbase processes running interbase,
is this right? I thought SS ran as one single process?

I've even marked /xinet.d/interbase as disabled
so I just don't get it.

Please help,
t eriksson

[Firebird-test] Testing on Win systems

[Martijn T. <m.t...@up...>]

Hi all,

I remember something about a testing suite that can be run on a Windows
system without having some C compiler installed, right? If so, I would like
to get going with it... I've a seperate machine now to do that stuff :)


Let me know,

--
Martijn Tonies
InterBase Workbench - the developer tool for InterBase and Firebird
http://www.interbaseworkbench.com

Upscene Productions
http://www.upscene.com

"This is an object-oriented system.
If we change anything, the users object."

[Firebird-test] Grant - SQLCODE = -607

[Konstantin B. <ko...@sp...>]

Hi!!!

I install FR 6.2.608
I Run My SqlScript
On the Grant Errors:

GRANT Select, Update(TALKSKEEP, KEEPALWAYS, ADDINARCHIV) ON Task  TO
MagArchiv;
Statement failed, SQLCODE = -607

unsuccessful metadata update
-STORE RDB$USER_PRIVILEGES failed in grant
-message length error (encountered 32, expected 64)
GRANT References(Code) ON rTiePlace     TO MagArchiv;
Statement failed, SQLCODE = -607

unsuccessful metadata update
-STORE RDB$USER_PRIVILEGES failed in grant
-message length error (encountered 32, expected 64)
GRANT References(Code) ON RTIEPLACE     TO MagArchiv;
Statement failed, SQLCODE = -607

unsuccessful metadata update
-STORE RDB$USER_PRIVILEGES failed in grant
-message length error (encountered 32, expected 64)


Why there was this mistake and as from it to get rid?

--
    Best regards, Konstantin.





--
    Best regards, Konstantin.

* Origin: Novosibirsk, Russia (FidoNet 2:5000/71.121)
e-mail Home: KB...@p1...     Work: ko...@sp...
ICQ:   48750138

[Firebird-test] Updated gtcs.gbk

[Neil M. <nm...@zi...>]

From cvs log:

revision 1.7
date: 2001/11/25 23:57:34;  author: nmcc;  state: Exp;  lines: +158 -152
C_SQL_JOIN_1  C_SQL_JOIN_41  IDML_C_SUBQ_06  PROC_ISQL_11
Updated to remove abiguous field warnings.

PROC_CAST11_ISQL
CV has changed "conversion error from string" message to have correct
contents - changed output expected to reflect this.


-- 
Neil McCalden                @home     nm...@zi...     

[Firebird-test] Engine profiling and performance testing

[Leyne, S. <sl...@at...>]

All,

Mike N's conversation regarding FB2 threads/design/etc. is great!

I realized, however, that we are all (myself included) starting to
suggest solutions to a problem(S) which is not clearly defined.=20

Namely, what are *the* bottlenecks in the current engine design?

I'm not trying to suggest that the ideas put forward by Mike (or anybody
else) are not good or won't be ultimately adopted.  The problem,
however, is that we are 'guessing' at the source of the real problems.

So the question is then, how can we profile the engine in order to
determine where the performance 'sink holes' are?

Related to the issue of profiling is performance testing/baselines.

Also, I think we need to establish some baseline values for the engine,
which can then be used to judge the impact of any change made to the
engine/code. =20

We can't be working on 'gut' senses of performance -- look at the
ongoing debate about cache size > 10000 pages.  We need imperical data
to backup our assertions (FB vX is 100% faster than IB 7.0 -- work with
me here :-] )

I know that Tord Hammer has the TPC-R tests to running with the engine.


There are, however, a couple of issues:

- The TPC-R tests, are more "single user" tests unlike the TPC-C which
are multi-user.

- The TPC-R tests currently don't run on Windows.

So, while we can use these tests to benchmark such things as the
performance impact of page size and/or cache size, they aren't can't
mimic a "real life"/multi-user environment.

So the question the here is, what can we do to get some 'multi-user'
tests?


I'd be willing to dedicate a server box to allow for the testing to be
performed and a baseline established, and to keep this box unchanged (a
known/stable configuration) for as long as we need.  I also have a
number of Windows desktops which I could use for the necessary
"clients".

Finally, who would be interested in working on these problems?


Sean

[Firebird-test] FYI: Atkin newsserver will be unavailable on Sunday Sept. 9th 00:01-01:00 EDT

[Sean L. <sl...@at...>]

I have been advised by my ISP that they will be performing equipment service
during this period, which will result in loss of service.

Therefore the server will not be available.


Sean Leyne
Atkin & Associates

[Firebird-test] FYI: Atkin newsserver will be unavailable on Sunday Sept. 9th 00:01-01:00 EDT

[Sean L. <sl...@at...>]

I have been advised by my ISP that they will be performing equipment service
during this period, which will result in loss of service.

Therefore the server will not be available.


Sean Leyne
Atkin & Associates

Re: [Firebird-test] Testing Firbird --- SI-

[Mark O'D. <mar...@lu...>]

Konstantin Kuznetsov wrote:

>>however you think it should work.
>>
>Mark excuse me for stupid question.
>How I can use this tag in future?
>

If at a later stage you want to rebuild what you had at this time, you can use the -r option to check out the exact version. as in:

cvs checkout -r T1_0_0Beta2-linux-T1_0_0-0a_Beta2  interbase


This can also be applied to checking for differences from beta to 
current via:

cvs diff -r T1_0_0Beta2-linux-T1_0_0-0a_Beta2 .



There is also a option to create branch tags, so that edits can be 
applied independatly to the main path, but thats for maintenance 
releases vs continued development.

There are quite a few more tricks as well, but the checkout and diff are 
the main uses.


Cheers

Mark

-- 
Your database needs YOU!
http://www.firebirdSQL.org

Re: [Firebird-test] Testing Firbird --- SI-

[Konstantin K. <klk...@ns...>]

Hi Mark!
>
> Neils change I believe has just been committed, (perhaps he or Claudio
would want to clarify it). If you can incorperate it (by just updating the
file) jrd/idx.h.  and give it a quick rerun through the TCS that would be
best.
Ok I move to Neil`s version 384.
And retest it all. I think it will be in 3-5 days.
>
> Failing that, don't worry too much about it, this is a beta and the error
as I understand it is not fatal and having it would match the current win32
and linux beta2 release. Since we have already a number of other changes
afoot: skip(x) offset value change; produce errors for ambigious field
references; etc, I'd say don't worry too much about it, as I have a
suspicion that in a weeks time we will be knocking on your door again asking
for a rebuild for beta3 or RC1 or whatever we call our next version :-),
>
Your are welcome  ...
>
> I think you got the message (Im guessing from the post) about the tag
> structure, but just in case you missed it:
>
> I was using the format:
> T1_0_0Beta2-linux-T1_0_0-0a_Beta2
>
> The format being
> <version> - <platform> - <something to match the physical distrib.
> package name>
>
> applied via:
>
> cvs -z9 tag T1_0_0Beta2-linux-T1_0_0-0a_Beta2 .
>
I will use T1_0_0Beta2-Solaris-X86-T1_0_0-<reported version by -z>
>
> Im not sure how your i386 tag should differ from Neils, sparc tag, be
> that as part of the <package> bit or the <platform> bit, but Im happy
> however you think it should work.
Mark excuse me for stupid question.
How I can use this tag in future?

Konstantin.

Re: [Firebird-test] How to make Sean feel like an idiot...

[John B. <bel...@cs...>]

Mark,

On Monday, August 27, 2001, at 09:28 PM, Mark O'Donohue wrote:

> Leyne, Sean wrote:
>
>> How to make Sean feel like an idiot in 3 easy steps...
>>
>> {Fuck!!!! Somebody just shoot me and but me out
>> of my misery!}
>>
>
> Ok let me revise and move some of the details around based on our new 
> configuration.
>
> Previously I got the CF_ISQL series to run with minimal trouble, and 
> they all passed.  I did modify some files to point to win32 specific 
> stuff, but this was early days, and the instructions were for unix.
> The tests that need to compile code I didn't try, in principle there 
> should not be too many problems, since we spawn a shell and I even ran 
> the MS Visual c++ 6.0 compiler but I didn't go too far, mainly i wanted 
> to see if it was possible.
>
> To run it once the environment variables point towards the right 
> ISQL.EXE file etc, it's really just a matter of typing in $make or 
> $runtest.
>
> But wait a bit (a day or so, for me to get time to rewrite the notes 
> and to give it a rough try - Im a bit busy currently), and with a 
> little bit of help, I feel we'll be able to get it working like a charm.
>
> [...]

I just committed the set of files needed to get autoconf working with 
the dejagnu testing stuff.  It appears to work well on my machine at 
home.  I'm sure it (configure.in, etc) will need a little tweaking to 
get right, but I think in the long run having something autoconf based 
will make handling different install situations much easier.

-John

RE: [Firebird-test] How to make Sean feel like an idiot...

[Leyne, S. <sl...@at...>]

Mark,

> The tests that need to compile code I didn't try, in principle there 
> should not be too many problems, since we spawn a shell and I 
> even ran 
> the MS Visual c++ 6.0 compiler but I didn't go too far, 
> mainly i wanted 
> to see if it was possible.

Would it be difficult to ship pre-compiled C test for the Windows
platform, thus bypassing the entire compile issue.  This would provide
for someone to run all the tests, but would not represent a TCS run as
far as a release would be concerned -- think of it as a 'nightly' build
check version of TCS.


> But wait a bit (a day or so, for me to get time to rewrite 
> the notes and 
> to give it a rough try - Im a bit busy currently), and with a 
> little bit 
> of help, I feel we'll be able to get it working like a charm.

Hey, I'm a patient man, take three days!


> In terms of using the gcc compiler, for testing, It would be 
> nice if we 
> can, but I think we will need to experiment with it a bit, I've only 
> played with cgywin gcc to compile 4-5 line programs, and 
> although it's 
> as good as the linux version, I have no idea how it interacts 
> with the 
> MS world.  I like the idea of compiling fb using gcc on 
> cygwin,  but I don't have time to try it out.

Something that we can look at later.


> And I'd better brush up on my spanish :-).

Muchos gracias!


Sean

Re: [Firebird-test] How to make Sean feel like an idiot...

[Mark O'D. <mar...@lu...>]

Leyne, Sean wrote:

>How to make Sean feel like an idiot in 3 easy steps...
>
>{Fuck!!!! Somebody just shoot me and but me out
>of my misery!}
>

Ok let me revise and move some of the details around based on our new 
configuration.

Previously I got the CF_ISQL series to run with minimal trouble, and 
they all passed.  I did modify some files to point to win32 specific 
stuff, but this was early days, and the instructions were for unix.  

The tests that need to compile code I didn't try, in principle there 
should not be too many problems, since we spawn a shell and I even ran 
the MS Visual c++ 6.0 compiler but I didn't go too far, mainly i wanted 
to see if it was possible.

To run it once the environment variables point towards the right 
ISQL.EXE file etc, it's really just a matter of typing in $make or $runtest.

But wait a bit (a day or so, for me to get time to rewrite the notes and 
to give it a rough try - Im a bit busy currently), and with a little bit 
of help, I feel we'll be able to get it working like a charm.

In terms of using the gcc compiler, for testing, It would be nice if we 
can, but I think we will need to experiment with it a bit, I've only 
played with cgywin gcc to compile 4-5 line programs, and although it's 
as good as the linux version, I have no idea how it interacts with the 
MS world.  I like the idea of compiling fb using gcc on cygwin,  but I 
don't have time to try it out.

And I'd better brush up on my spanish :-).

Cheers

Mark

-- 
Your database needs YOU!
http://www.firebirdSQL.org

Re: [Firebird-test] How to make Sean feel like an idiot...

[John B. <bel...@cs...>]

Sean,

On Monday, August 27, 2001, at 03:57 PM, Leyne, Sean wrote:

> John,
>
>>> How to make Sean feel like an idiot in 3 easy steps...
>>
>> That was not my intent when I committed the new code, but... ;-)
>
> I realized that this was not the intent, I was just relating the effect.
>
>> Here is a quick run-down of what needs to be done for unsupported
>> platforms:
>
> Unsupported platforms?

Maybe unintegrated platforms is a better term.  It works on the 
platforms, just not seamlessly yet.

-John

RE: [Firebird-test] How to make Sean feel like an idiot...

[Leyne, S. <sl...@at...>]

John,

> > How to make Sean feel like an idiot in 3 easy steps...
> 
> That was not my intent when I committed the new code, but... ;-)

I realized that this was not the intent, I was just relating the effect.

> Here is a quick run-down of what needs to be done for unsupported 
> platforms:

Unsupported platforms?


Sean

Re: [Firebird-test] How to make Sean feel like an idiot...

[John B. <bel...@cs...>]

Sean,

On Monday, August 27, 2001, at 12:45 PM, Leyne, Sean wrote:

>
> How to make Sean feel like an idiot in 3 easy steps...

That was not my intent when I committed the new code, but... ;-)

>
> Step 1.	Have Sean download and install Cygwin
> 		(God! this is painfull process -- download 50mb,
> resulting in a 200 MB install!).
>
> 			{Ok, this is done}
>
> Step 2.	Have Sean download the latest TCS sources from SF.
>
> 			{Ok, I've fought that battle before and fell
> relatively confident -- it's done}
>
> Step 3.	Have Sean then try to figure out how to use the Cygwin bash
> shell to run John B's latest testsuite!
> 		(I read Mark's README file -- I would have had better
> luck reading a message from Claudio written in Spanish!)

I think we could arrange that, if you like :-)

>
> 			{Fuck!!!! Somebody just shoot me and but me out
> of my misery!}

The landscape has changed a lot from when Mark wrote the README.  Most 
of it is rubbish now. , especially the part that says what commands to 
run!  Anyway, since I don't have access to a cygwin install someone else 
(maybe Mark?) will have to help me get everything integrated correctly.  
Here is a quick run-down of what needs to be done for unsupported 
platforms:

1.  Export the correct BP.<platform> and ENV.<platform> blobs from the 
tcs platform DB.
2.  Tell tcs.exp where to find all the important utilities:
	isql
	gbak
	gpre
	gdef (I think it is used)
	drop_gdb (huh, whats this you ask?  A c API program included in TCS 
that needs to be compiled on the platform)
3.  Set the environment variables to point to the fb install.  I think 
this is handled in win32 via the registry.
4.  Set the WHERE_* pointers correctly.  There will be differences here 
depending on what server (classic/ss) is being tested.

Each of the above will be different on each platform, so a generic 
method of determining the correct settings is needed.  My current 
thoughts are create an autoconf script to generate a makefile that has 
all the paths, etc. in it.  Then have the makefile pass the info to the 
test scripts via environment variables.  So if you want to run TCS on a 
preexisting install you just have to do a ./configure; make.  We could 
integrate the configure aspect of the tests into the preexisting fb2 
build process in such a way that all fb2 builds can be tested out of the 
build directory without install.  The one exception might be win32, but 
if we put our minds to it I'm sure we can come up with something.

-John

[Firebird-test] New gtcs.gbk - v1.6

[Neil M. <nm...@zi...>]

I have just checked in a new version of gtcs.gbk which has the following 
two changes:-

C_SQL_JOIN_4
 Fix inconsistency in query sort order and output I introduced in an
 earlier version.

DSQL_DOMAIN_21 & DSQL_DOMAIN_22
 Fix output to match output produced by Beta2 since CVC fixed
 [ firebird-Bugs-448062 ] ALTER DOMAIN leaves CONSTRAINT word


The problem of C_DSQL_RI_ALT_ADD failing has been fixed by the change
reverting an earlier edit I checked in to the main tree this morning. 

I have not updated the output DSQL_DOMAIN_08 so it still also reports as
a failure but it is just different line numbers being reported by
Claudio's changes. The line number seems incorrect depending on if you
count comments and blank lines so it may change again.


-- 
Neil McCalden                @home     nm...@zi...     

[Firebird-test] How to make Sean feel like an idiot...

[Leyne, S. <sl...@at...>]

How to make Sean feel like an idiot in 3 easy steps...

Step 1.	Have Sean download and install Cygwin 
		(God! this is painfull process -- download 50mb,
resulting in a 200 MB install!).

			{Ok, this is done}

Step 2.	Have Sean download the latest TCS sources from SF.  

			{Ok, I've fought that battle before and fell
relatively confident -- it's done}

Step 3.	Have Sean then try to figure out how to use the Cygwin bash
shell to run John B's latest testsuite!  
		(I read Mark's README file -- I would have had better
luck reading a message from Claudio written in Spanish!)

			{Fuck!!!! Somebody just shoot me and but me out
of my misery!}


Sean

Re: [Firebird-test] Testing Firbird

[John B. <bel...@cs...>]

Sean,

On Monday, August 27, 2001, at 11:12 AM, Leyne, Sean wrote:

> John,
>
> I am very pleased to hear of your progress with the new TCS suite.

But my platform support has suffered, along with the bug fixes I should 
be doing :-(

>
> I appreciate that we will probably never be able to elimate the need for
> cygwin on Windows, and can live with that.
>
> The fact that the new suite does not require the database to be running
> and provides the hope of being able to run the tests without a compiler
> and to test an engine build is a FANTASTIC step!

The next question is does cygwin come with the gcc compiler?  If so just 
installing cygwin may provide sufficient tools to run the full test 
suite.  Although I don't know if we want the added dependency.

>
> Now all I need is a set of installation/configuration instructions (TCS
> for Dummies) and a zip/tarball of the ISQL sub series...  then this
> 'idiot' can begin to run the tests and work with others to expand there
> scope.

Well, all the files are in the TCS module, in the deja-testsuite 
directory.  I haven't combed through them to figure out which ones 
require a compiler and which ones don't.  Likely candidates for one that 
do are all the tests that start with "C_", for example "C_SQL_JOIN".  As 
for the instructions I can provide you with the Darwin 
instructions :-).  I don't know how to make it work on win32.  I don't 
even have access to a win32 machine (and I prefer to keep it that 
way :-).  Mark O. had an earlier version of tcs.exp working on win32, so 
for the time being he will be a better resource for that effort.

-John

RE: [Firebird-test] Testing Firbird --- SI-T1.0.0.366 Firebird Be

[Leyne, S. <sl...@at...>]

John,

I am very pleased to hear of your progress with the new TCS suite.

I appreciate that we will probably never be able to elimate the need for
cygwin on Windows, and can live with that.

The fact that the new suite does not require the database to be running
and provides the hope of being able to run the tests without a compiler
and to test an engine build is a FANTASTIC step!

Now all I need is a set of installation/configuration instructions (TCS
for Dummies) and a zip/tarball of the ISQL sub series...  then this
'idiot' can begin to run the tests and work with others to expand there
scope. 


Sean

Re: [Firebird-test] Testing Firbird --- SI-T1.0.0.373 now

[Mark O'D. <mar...@lu...>]

Konstantin Kuznetsov wrote:

>Hi All and Mark !
>
>[very big skip]
>
>>I extracted the data using Franks extract kit, and then ran these using
>>the trial dejagnu test kit, which made it a little easier to get at the
>>intermediate files.
>>
>>So all in all, your looking pretty good, there :-).
>>
>Thanks Mark for your help and explanation !
>
>If so -> look good !
>Now I have compiled 1.0.0.373 Classic and Super, wich are now  bouth passed
>throu TCS with the same as 366 results.
>I ready to release something, tag my sources or wait for Neil improvement
>of TCS database.
>Your's advices ?
>
>This thanks Konstantin Kuznetsov
>
Hi Konstantin

 
Congratulations, it's a long haul, :-) 

Neils change I believe has just been committed, (perhaps he or Claudio would want to clarify it). If you can incorperate it (by just updating the file) jrd/idx.h.  and give it a quick rerun through the TCS that would be best.

Failing that, don't worry too much about it, this is a beta and the error as I understand it is not fatal and having it would match the current win32 and linux beta2 release. Since we have already a number of other changes afoot: skip(x) offset value change; produce errors for ambigious field references; etc, I'd say don't worry too much about it, as I have a suspicion that in a weeks time we will be knocking on your door again asking for a rebuild for beta3 or RC1 or whatever we call our next version :-),


I think you got the message (Im guessing from the post) about the tag 
structure, but just in case you missed it:

I was using the format:
T1_0_0Beta2-linux-T1_0_0-0a_Beta2

The format being
<version> - <platform> - <something to match the physical distrib. 
package name>

applied via:

cvs -z9 tag T1_0_0Beta2-linux-T1_0_0-0a_Beta2 .


Im not sure how your i386 tag should differ from Neils, sparc tag, be 
that as part of the <package> bit or the <platform> bit, but Im happy 
however you think it should work.


Cheers

Mark


-- 
Your database needs YOU!
http://www.firebirdSQL.org 

Re: [Firebird-test] Testing Firbird --- SI-T1.0.0.373 now

[Konstantin K. <klk...@ns...>]

Hi All and Mark !
>
[very big skip]
> I extracted the data using Franks extract kit, and then ran these using
> the trial dejagnu test kit, which made it a little easier to get at the
> intermediate files.
>
> So all in all, your looking pretty good, there :-).
>
Thanks Mark for your help and explanation !

If so -> look good !
Now I have compiled 1.0.0.373 Classic and Super, wich are now  bouth passed
throu TCS with the same as 366 results.
I ready to release something, tag my sources or wait for Neil improvement
of TCS database.
Your's advices ?

This thanks Konstantin Kuznetsov

Re: [Firebird-test] Testing Firbird --- SI-T1.0.0.366 Firebird Be

[John B. <bel...@cs...>]

Hi,

On Sunday, August 26, 2001, at 08:44 PM, Mark O'Donohue wrote:

>
> Hi Sean
>
> Leyne, Sean wrote:
>
>>
>> What would it take to have the reports produce a more 'intelligible'
>> output?
>>
>> This output seems absolutely useless!
>>
> It is, what John is working on will enable us to do a unified diff (aka 
> the code changes), and to mix the stdout/stderror streams so the output 
> appears normal and in context.
> It's not far off, last report was there were only 8 TCS tests that when 
> converted still produced different results to the TCS run - but I 
> should let him describe that.

I have just committed (as anyone on firebird-checkins can attest to :-) 
a whole new test suite.  Most of it consisted of the flatfile data 
extracted from TCS, but there was an updated test script in there to.  
Right now my latest build passed all but 2 or 3 tests using the new test 
suite.  Here is a [maybe not so] brief description of the suite:

It is a straight lift of TCS.  I have not modified any of the tests or 
the test data itself.  I'm running of the same blob's used in TCS, 
except they are in flat files now.  What the test script does is emulate 
(to the best of my TCS knowledge) all aspects of the TCS pattern 
replacement, test script construction, and data file construction 
process.

Once the test data has been extracted and the environment configured 
properly the test is run.  The output of the test is captured in the 
same order a user would see it (stdout/stderr intermixed).  The current 
TCS expected output is in the format stderr first, then stdout.  This is 
one of the things that makes determining where a test failed difficult.  
As the new tcs.exp test reads the lines (one at a time) it matches them 
against the original TCS expected output to check for failure.  A fail 
(expected or unexpected) is reported.  The code also handles expected 
output files with intermixed stdout/stderr.  This format is what Mark 
was talking about in the quoted message.  After we are satisfied with 
the correctness of tcs.exp we can use it to generate new intermixed 
expected output files that will be easier to understand.

Current tcs.exp does not contain any support for creating new tests (ie, 
capturing the output).  That support needs to be added, but it won't be 
very difficult.

tcs.exp also lacks the correct hooks into the build/install 
environment.  That means it only works on my machine :-)  But the good 
news is all it takes to get working on another machine is correctly 
setting a couple of environment variables.  Tighter integration with the 
build process (something obtainable for FB2, but probably not worth the 
effort for FB1) can allow full tcs.exp testing against a built engine 
without installing it and while another engine is still running on the 
same host!

tcs.exp is written in Tcl using expect and dejagnu.  These programs are 
all available for win32 (Mark had an earlier version of tcs.exp running 
in win32 under vmware) and available from cygwin.  I'm not sure how 
feasible installing pure win32 ports the the programs is.  It is quite 
possible the tcs tests assume the presence of a program (for example sh) 
that tcs.exp doesn't explicitly rely on.

tcs.exp still requires a compiler to be present to test.  But it also 
has the notion of sub series.  Using sub series we could define a set of 
tests that doesn't rely on the compiler (for example the isql tests).  
These tests could be run on an end-users machine assuming they has 
cygwin installed (for the afore mentioned dependencies).  They wouldn't 
need the MS (or whomever) compiler.

Well, I think I've written enough for the time being...

-John