RE: [Firebird-test] Catching up on postings

[David J. <dav...@ea...>]

Hi,

First, I want to apologize for the not very pleasant tone of my last
message--sorry.

Here are a few thoughts interspersed
On 2001.05.14 09:49:47 -0400 "Leyne, Sean" wrote:
> David,
> 
> -----Original Message-----
> From: 	David Jencks [mailto:dav...@ea...] 
> Sent:	Monday, May 14, 2001 1:26 AM
> To:	fir...@li...
> Subject:	Re: [Firebird-test] Catching up on postings
> 
> Hi,
> 
> Well, yes, I already started talking about it.
> 
> <Sean>
> I didn't mean to imply that you hadn't.  I just wanted to re-focus the
> discussion to a higher level before we got immersed in the almost
> inevitable "religious" wars regarding C++ vs. Java vs. Python...
> 
> "Talking about the tools before we have defined the problem is like
> putting the cart before the horse."
> </Sean>
<david>
Well, perhaps, but there is also the xprogramming idea of implementing the
simplest possible functionality for the most important use, then adding
more for the next most important use while refactoring to improve the
design,....  The first version of what I am working on I wrote to (1) have
a java based scripting method and (2) test the interclient changes I made a
while back.  I'm now rewriting it to be independent of ant and provide
easier to use regression tests.
</david>
> 
> 
> I have nearly completed a java scripting/testing solution based on xml
> script/test files.  It can include the results of db access in the xml
> file, and compare the current results with a stored reference.  Being
> written in java, it will be easy to extend to test parallel db access.
> 
> <Sean>
> That's great news! 
> 
> I'm wondering though, why did you choose XML as the file format, as
> compared to straight text?
> </Sean>
> 
<david>
XML structure/parsers give you the ability to find just what you want in an
arbitrarily large document without having to write a parser.  Navigation is
also plausible.  It should be very easy to provide a nice html views of
resultsets from the xml translation.  The xml structure makes it easy to
pinpoint where differences between the old and new results are... give a
path to  the elements.  A big reason though is probably because this
started life as an ant task, and ant build files are xml based.
</david>
>  
> Obviously it is best adapted to testing via jdbc drivers, 
> 
> <Sean>
> Have you looked at the Sun JDBC compliance testing routines?
> </Sean>
> 
<david> Only marginally... they mostly test the jdbc driver rather than db
functionality as far as I know</david>
> but I think it might be worthwile investigating how hard it would be to
> write adapters to
> use odbc drivers, IBO, and possibly generating dynamic and static sql c
> or
> c++ programs.  
> 
> <Sean>
> Personally, I'd like to leave the issue of IBO and dynamic/static C or
> C++ programs to a much, much later time.  We have enough on our plate.
> </Sean>
> 
<david> It did occur to me that at least some c client functionality might
be testable with exactly the same code by using the much reviled jdbc-odbc
bridge and Jim's odbc driver.</david>
> I think having a single cross platform execution engine would be rather
> desirable.  
> 
> <Sean>
> Agreed!
> </Sean>
> 
> I don't think reliance on isql is appropriate.
> 
> <Sean>
> Before Ann has a chance to ask, why not?  
> 
> ISQL needs to be tested, so why not use it as part of the tests?
> </Sean>

<david>I suppose you are right that it needs testing.  I have not had good
results with isql, maybe I don't know how to make it commit.  It doesn't
usually show me what's in the database, put there by interclient.  Also as
far as I know you can't test parameterized queries with it. Testing
threading/parallel execution is as far as I know impossible. It seems to me
that testing the ways client programs will use the db is more important
than testing a scripting engine.  So for my money, the most important
testing is through jdbc drivers, odbc drivers, ibo, ibx, bde, c code that
uses the dynamic sql interface, static sql, and gdml, in approximately that
order (first 3 can be shuffled).</david>
> 
> 
> 
> Translating the TCS tests to any other format is going to be a large
> amount
> of work.  Its easy enough to dump say each test into an xml file, but
> extracting the sql and expected results into a more reasonable format
> will
> be time consuming.  The NIST tests look only slightly easier to move to
> a
> different system than TCS.
> 
> <Sean>
> Don't understand what you mean about extracting the SQL?
> </Sean>
<david>
Most of the TCS I looked at(not very much) consisted of a simple sql
statement plus a c program to run it and produce text output: elsewhere in
TCS is the expected output.  The format of the output is highly dependent
on the actual c program used.  To me, the important parts are in order (1)
the SQL (2) the logical structure and contents of the results (3) the
logical setup information (4) the c program (5) the text layout of the
results.

Finding the SQL and putting it into the xml format I am proposing is fairly
easy.  Finding the expected results is a little harder.  I don't know how
to automatically translate the expected results into the xml result set
format I propose.  I think someone has to look at it and decide if the
meaning is the same.  I suppose it might be possible to take the xml and
transform it into text of the same format as TCS, and compare that...I'll
have to think about this some more.
</david>
> 
> Would a provisional dtd for what I am working on and sample document be
> helpful? Both are subject to change by me before I show the code, but
> should give an idea of what I'm after.
> 
> <Sean>
> Personally, I'd like to talk about general approach before diving into
> the details.
> </Sean>
> 
<david>
Here are some things I think are important or a good idea.

1. separate logical content of tests and results from formatting details.
2. Separate platform/execution context from content of test.
3. Concentrate on testing drivers used by application programs.
4. Have as much of the testing framework work on all platforms as possible
with as little modification as possible.
5. File based version control.
6. test suite updates available in small pieces. ( not the whole tcs db)

I hope you can comment on my samples posted in a different message.
Thanks!
</david>


> 
> _______________________________________________
> Firebird-test mailing list
> Fir...@li...
> http://lists.sourceforge.net/lists/listinfo/firebird-test
>