RE: [Firebird-test] Testing FB1

[Leyne, S. <sl...@at...>]

John,

> -----Original Message-----
> From: John Bellardo [mailto:bel...@cs...]
> Sent: Friday, August 03, 2001 1:29 PM

...

> I was talking about the tests themselves.  Currently there are two 
> levels in TCS, global and local.  The global database (gtcs.gdb) 
> contains tests and other configuration information that is 
> common across 
> all platforms.  The local database (ltcs.gdb, a sym. link to 
> the correct 
> platform db) contains tests and configurations that override those in 
> the global database.

My apologizes, I didn't get that from your original message.

Would storing the data in different directories solve the problem?

I thinking of something along the lines of the new directory structure
for build info for the various ports.

maybe something like:

	TCS\
		Environment\
		Global\
			gpre\
			SQL\
			ODBC\
			Java\
			...
		Local\
			{port}\
				gpre\
				SQL\
				ODBC\
				Java\
				...			
			...\
		Results\
		Expected\
		

Another option, a much better one (now that I think of it), use a single
XML file in which both the Global/Default and Local information is
stored, in that way all the data about a test (or template or
boilerplate) is in one place, while supporting local/port specific
versions as required.  The layout of the file could go something like:

	<XML>
		<Description>
		...
		</Description>
		<Script Platform = "All">
		...
		</Script>
		<Script Platform = "Linux">
		...
		</Script>
	</XML>

> Which brings me to my next question.  How much of the existing TCS 
> functionality to we want to maintain?  There are a number of features 
> that I have never used, but are there.  For example all the test 
> failures are stored back into the local test database.
> Do you want to move the boiler plates and environments out of 
> TCS too?  

Yep!

> What is the motivation for moving to flat files?  They are easier to 
> track in CVS and to change.  But moving all the test files and test 
> names to flat file would address that.  We don't have to move 
> *everything* out of the databases.

Well, I do.  It seems illogical for you to need to have the database
engine running, in order to test the database engine.  If there is a
failure in the test, is it due to the fact that the engine running the
test failed or did the testing program fail when updating the testing
data.

With respect to the templates and environments, etc...  I far as I can
tell, all this information is 'key' to the operation of the TCS system
as such, we need to ensure that all of this data is exposed and easily
available to users to review and change as appropriate.

Finally, in a closed product it's fine for all the testing information
to be hidden, in an open product the testing process/programs need to be
completely open/available for review.  Using the database only hides the
information.

> TCS is combersome to use, and a replacement that allowed us (unix 
> guys :-) to just run "make check" or "make certify" would be 
> great.  But 
> where do we draw the line between "improving" the existing, 
> functional 
> test program and working on the engine?

Unfortunately, when dealing with a very complex program (as FB is today)
its reliability becomes almost more important than the scope of the
functionality.

Think of it.

What would you think if a database that had been running for 2 months
stopped working after you did an update?

If that happended a couple of times, you'd probably look for a new
product and tell everybody who asked, that the product was a real piece
of s**t!

It's one thing to have to have to work around missing functionality but
if you can't depend on the existing functionality!

Bottom line is that we need to spend time working on the testing
programs/functions if we really expect to be able to move forward with
the engine.  

Currently we are working with the existing code and making medium level
(in a few cases) changes to the engine.  

When we start doing 'real' work on v2, it will involve much much more
serious changes, if we can't provide some guarantee about our work (i.e.
this has completely passed our QA testing) -- Who will use the product
in a production environment?

I realize that this will not make some people happy, having to work on
improving/changing TCS, but this is simply the fact of life -- it's not
glamorous, in fact it's kind of tedious.  But it is ABSOLUTELY
necessary.


Sean