firebird-test

RE: [Firebird-test] Catching up on postings

[Leyne, S. <sl...@at...>]

-----Original Message-----
From: 	Jason Chapman (JAC2) [mailto:ja...@ja...] 
Sent:	Monday, May 14, 2001 6:28 AM
To:	fir...@li...
Subject:	Re: [Firebird-test] Catching up on postings


<Snip>

> 3) All of the testing programs (including any script editors)
> should portable, to run on any platform which the engine runs on.
This may hamper development, I could work on some bits in Delphi for
Win32,
but that would under the "must be under all OS'" be a no-no.

<Sean>
Don't get me wrong, I'm one of Bill Gates drones, I know practically
nothing about C, C++ or *nux so I'd love to say that we should write the
testing programs in Delphi/Kylix.  However, the frank reality is that
the testing programs need to apply much more broadly.
</Sean>

> 4) Any choice of development tools (Delphi, C++, Python, Java or MS
> C# <big grin>) should carefully consider the overall expertise of the
> project members.
3 damages 4. Suddenly we've lost Delphi + MS anything.

<Sean>
Keep in mind that the testing programs are really designed (even TCS
today) as a command processor; taking in commands (scripts), executing
the commands and then reporting the results.  So, in the long run most
of the work in the testing process will be in the management/creation of
testing scenarios/scripts, which is something that everybody can make
very important contributions.

The testing programs themselves will probably require little maintenance
of their own, so the fact that most of us would really feel more
comfortable using Delphi which isn't cross-platform should really matter
in the long run - there will always be someone who can work on the
programs for us <grin>.

There, however, could be a role for Delphi in creating a nifty script
editor/manager, to save use from the depths/details of the XML (to use
David's example).
</Sean>

RE: [Firebird-test] Catching up on postings

[Leyne, S. <sl...@at...>]

David,

-----Original Message-----
From: 	David Jencks [mailto:dav...@ea...] 
Sent:	Monday, May 14, 2001 1:26 AM
To:	fir...@li...
Subject:	Re: [Firebird-test] Catching up on postings

Hi,

Well, yes, I already started talking about it.

<Sean>
I didn't mean to imply that you hadn't.  I just wanted to re-focus the
discussion to a higher level before we got immersed in the almost
inevitable "religious" wars regarding C++ vs. Java vs. Python...

"Talking about the tools before we have defined the problem is like
putting the cart before the horse."
</Sean>


I have nearly completed a java scripting/testing solution based on xml
script/test files.  It can include the results of db access in the xml
file, and compare the current results with a stored reference.  Being
written in java, it will be easy to extend to test parallel db access.

<Sean>
That's great news! 

I'm wondering though, why did you choose XML as the file format, as
compared to straight text?
</Sean>

 
Obviously it is best adapted to testing via jdbc drivers, 

<Sean>
Have you looked at the Sun JDBC compliance testing routines?
</Sean>

but I think it might be worthwile investigating how hard it would be to
write adapters to
use odbc drivers, IBO, and possibly generating dynamic and static sql c
or
c++ programs.  

<Sean>
Personally, I'd like to leave the issue of IBO and dynamic/static C or
C++ programs to a much, much later time.  We have enough on our plate.
</Sean>

I think having a single cross platform execution engine would be rather
desirable.  

<Sean>
Agreed!
</Sean>

I don't think reliance on isql is appropriate.

<Sean>
Before Ann has a chance to ask, why not?  

ISQL needs to be tested, so why not use it as part of the tests?
</Sean>



Translating the TCS tests to any other format is going to be a large
amount
of work.  Its easy enough to dump say each test into an xml file, but
extracting the sql and expected results into a more reasonable format
will
be time consuming.  The NIST tests look only slightly easier to move to
a
different system than TCS.

<Sean>
Don't understand what you mean about extracting the SQL?
</Sean>

Would a provisional dtd for what I am working on and sample document be
helpful? Both are subject to change by me before I show the code, but
should give an idea of what I'm after.

<Sean>
Personally, I'd like to talk about general approach before diving into
the details.
</Sean>

RE: [Firebird-test] Catching up on postings

[David J. <dav...@ea...>]

Hi,

First, I want to apologize for the not very pleasant tone of my last
message--sorry.

Here are a few thoughts interspersed
On 2001.05.14 09:49:47 -0400 "Leyne, Sean" wrote:
> David,
> 
> -----Original Message-----
> From: 	David Jencks [mailto:dav...@ea...] 
> Sent:	Monday, May 14, 2001 1:26 AM
> To:	fir...@li...
> Subject:	Re: [Firebird-test] Catching up on postings
> 
> Hi,
> 
> Well, yes, I already started talking about it.
> 
> <Sean>
> I didn't mean to imply that you hadn't.  I just wanted to re-focus the
> discussion to a higher level before we got immersed in the almost
> inevitable "religious" wars regarding C++ vs. Java vs. Python...
> 
> "Talking about the tools before we have defined the problem is like
> putting the cart before the horse."
> </Sean>
<david>
Well, perhaps, but there is also the xprogramming idea of implementing the
simplest possible functionality for the most important use, then adding
more for the next most important use while refactoring to improve the
design,....  The first version of what I am working on I wrote to (1) have
a java based scripting method and (2) test the interclient changes I made a
while back.  I'm now rewriting it to be independent of ant and provide
easier to use regression tests.
</david>
> 
> 
> I have nearly completed a java scripting/testing solution based on xml
> script/test files.  It can include the results of db access in the xml
> file, and compare the current results with a stored reference.  Being
> written in java, it will be easy to extend to test parallel db access.
> 
> <Sean>
> That's great news! 
> 
> I'm wondering though, why did you choose XML as the file format, as
> compared to straight text?
> </Sean>
> 
<david>
XML structure/parsers give you the ability to find just what you want in an
arbitrarily large document without having to write a parser.  Navigation is
also plausible.  It should be very easy to provide a nice html views of
resultsets from the xml translation.  The xml structure makes it easy to
pinpoint where differences between the old and new results are... give a
path to  the elements.  A big reason though is probably because this
started life as an ant task, and ant build files are xml based.
</david>
>  
> Obviously it is best adapted to testing via jdbc drivers, 
> 
> <Sean>
> Have you looked at the Sun JDBC compliance testing routines?
> </Sean>
> 
<david> Only marginally... they mostly test the jdbc driver rather than db
functionality as far as I know</david>
> but I think it might be worthwile investigating how hard it would be to
> write adapters to
> use odbc drivers, IBO, and possibly generating dynamic and static sql c
> or
> c++ programs.  
> 
> <Sean>
> Personally, I'd like to leave the issue of IBO and dynamic/static C or
> C++ programs to a much, much later time.  We have enough on our plate.
> </Sean>
> 
<david> It did occur to me that at least some c client functionality might
be testable with exactly the same code by using the much reviled jdbc-odbc
bridge and Jim's odbc driver.</david>
> I think having a single cross platform execution engine would be rather
> desirable.  
> 
> <Sean>
> Agreed!
> </Sean>
> 
> I don't think reliance on isql is appropriate.
> 
> <Sean>
> Before Ann has a chance to ask, why not?  
> 
> ISQL needs to be tested, so why not use it as part of the tests?
> </Sean>

<david>I suppose you are right that it needs testing.  I have not had good
results with isql, maybe I don't know how to make it commit.  It doesn't
usually show me what's in the database, put there by interclient.  Also as
far as I know you can't test parameterized queries with it. Testing
threading/parallel execution is as far as I know impossible. It seems to me
that testing the ways client programs will use the db is more important
than testing a scripting engine.  So for my money, the most important
testing is through jdbc drivers, odbc drivers, ibo, ibx, bde, c code that
uses the dynamic sql interface, static sql, and gdml, in approximately that
order (first 3 can be shuffled).</david>
> 
> 
> 
> Translating the TCS tests to any other format is going to be a large
> amount
> of work.  Its easy enough to dump say each test into an xml file, but
> extracting the sql and expected results into a more reasonable format
> will
> be time consuming.  The NIST tests look only slightly easier to move to
> a
> different system than TCS.
> 
> <Sean>
> Don't understand what you mean about extracting the SQL?
> </Sean>
<david>
Most of the TCS I looked at(not very much) consisted of a simple sql
statement plus a c program to run it and produce text output: elsewhere in
TCS is the expected output.  The format of the output is highly dependent
on the actual c program used.  To me, the important parts are in order (1)
the SQL (2) the logical structure and contents of the results (3) the
logical setup information (4) the c program (5) the text layout of the
results.

Finding the SQL and putting it into the xml format I am proposing is fairly
easy.  Finding the expected results is a little harder.  I don't know how
to automatically translate the expected results into the xml result set
format I propose.  I think someone has to look at it and decide if the
meaning is the same.  I suppose it might be possible to take the xml and
transform it into text of the same format as TCS, and compare that...I'll
have to think about this some more.
</david>
> 
> Would a provisional dtd for what I am working on and sample document be
> helpful? Both are subject to change by me before I show the code, but
> should give an idea of what I'm after.
> 
> <Sean>
> Personally, I'd like to talk about general approach before diving into
> the details.
> </Sean>
> 
<david>
Here are some things I think are important or a good idea.

1. separate logical content of tests and results from formatting details.
2. Separate platform/execution context from content of test.
3. Concentrate on testing drivers used by application programs.
4. Have as much of the testing framework work on all platforms as possible
with as little modification as possible.
5. File based version control.
6. test suite updates available in small pieces. ( not the whole tcs db)

I hope you can comment on my samples posted in a different message.
Thanks!
</david>


> 
> _______________________________________________
> Firebird-test mailing list
> Fir...@li...
> http://lists.sourceforge.net/lists/listinfo/firebird-test
> 

RE: [Firebird-test] Catching up on postings

[<pc...@at...>]

Hi,

On 15 May 2001, at 0:25, David Jencks wrote:

> Here are some things I think are important or a good idea.
> 
> 1. separate logical content of tests and results from formatting details.
> 2. Separate platform/execution context from content of test.
> 3. Concentrate on testing drivers used by application programs.
> 4. Have as much of the testing framework work on all platforms as possible
> with as little modification as possible.
> 5. File based version control.
> 6. test suite updates available in small pieces. ( not the whole tcs db)

I think that this is the best way to go. We would have test 
definitions (with data, executed/tested functionality and expected 
results) in portable, easy to manipulate format (XML). When test 
framework would be defined, it could be easily implemented in 
many languages, with different access methods and on any 
platform. 

Just a side note: We should separate the test database content 
XML from tests description XML (and reference it).

Actually, I *love* this idea more and more.

Best regards
-- Pavel

RE: [Firebird-test] Catching up on postings

[Leyne, S. <sl...@at...>]

David,

See my comments below:

> -----Original Message-----
> From: David Jencks [mailto:dav...@ea...]
> Sent: Tuesday, May 15, 2001 12:25 AM
> To: fir...@li...
> Subject: RE: [Firebird-test] Catching up on postings
> 
> > <Sean>
> > That's great news! 
> > 
> > I'm wondering though, why did you choose XML as the file format, as
> > compared to straight text?
> > </Sean>
> > 
> <david>
> XML structure/parsers give you the ability to find just what 
> you want in an 
> arbitrarily large document without having to write a parser.  
> Navigation is
> also plausible.  It should be very easy to provide a nice 
> html views of
> resultsets from the xml translation.  The xml structure makes 
> it easy to
> pinpoint where differences between the old and new results 
> are... give a
> path to  the elements.  A big reason though is probably because this
> started life as an ant task, and ant build files are xml based.
> </david>

<Sean>
David, are you proposing to store the test results in a separate file
from the original script or in the same file?

Personally, I think that the results should be stored separately.

I can appreciate the benefits of using XML for storing the results, it
would allow for a quick analysis of the elements to determine
discrepancies.

I must admit that I really prefer the simplicity of straight text for
the testing scripts themselves, perhaps I need to see a simple example
to appreciate the benefits of the structured layout.
</Sean>

> > 
> > I don't think reliance on isql is appropriate.
> > 
> > <Sean>
> > Before Ann has a chance to ask, why not?  
> > 
> > ISQL needs to be tested, so why not use it as part of the tests?
> > </Sean>
> 
> <david>I suppose you are right that it needs testing.  I have 
> not had good
> results with isql, maybe I don't know how to make it commit.  
> It doesn't
> usually show me what's in the database, put there by 
> interclient.  Also as
> far as I know you can't test parameterized queries with it. Testing
> threading/parallel execution is as far as I know impossible. 
> It seems to me
> that testing the ways client programs will use the db is more 
> important
> than testing a scripting engine.  So for my money, the most important
> testing is through jdbc drivers, odbc drivers, ibo, ibx, bde, 
> c code that
> uses the dynamic sql interface, static sql, and gdml, in 
> approximately that
> order (first 3 can be shuffled).</david>
> > 

<Sean>

Ah, there's your problem!  You're using InterClient! <grin>

The only problem I have with your points above is that the initial focus
is on testing the engine functionality.  

Using any data access layer (jdbc, odbc, IBO...) introduces into the
testing scheme possible 'artifacts'/errors related to the data access
layer and not engine functionality.  

The current ISQL approach virtually eliminated these type of errors,
saving testers from hunting down these 'false positive' results.
Furthermore, when it comes to diagnosing anomolies in data results, ISQL
has been used as the "gold standard" for determining whether a problem
is engine or data access layer related.

Don't get me wrong, I agree that data access layer testing is necessary
and should be considered as part of the overall goal.
</Sean>

> > <Sean>
> > Don't understand what you mean about extracting the SQL?
> > </Sean>
> <david>
> Most of the TCS I looked at(not very much) consisted of a simple sql
> statement plus a c program to run it and produce text output: 
> elsewhere in
> TCS is the expected output.  The format of the output is 
> highly dependent
> on the actual c program used.  To me, the important parts are 
> in order (1)
> the SQL (2) the logical structure and contents of the results (3) the
> logical setup information (4) the c program (5) the text layout of the
> results.
> 
> Finding the SQL and putting it into the xml format I am 
> proposing is fairly
> easy.  Finding the expected results is a little harder.  I 
> don't know how
> to automatically translate the expected results into the xml 
> result set
> format I propose.  I think someone has to look at it and decide if the
> meaning is the same.  I suppose it might be possible to take 
> the xml and
> transform it into text of the same format as TCS, and compare 
> that...I'll
> have to think about this some more.
> </david>

<Sean>
David, it was my understanding that the suite involved sending the
script commands to ISQL and then capturing the generated results.  I
wasn't aware of the associated c programs, so perhaps I (and others)
need an education about the functionality/implementation of the TCS
suite and the role of the C programs.

Accordingly, I _had_ been thinking it was actually fairly
straightforward to capture the results (although this may need some more
thought).

First, you successfully run the existing TCS suite on an existing engine
release (0.9-4).  This ensures that the engine is producing the expected
results.

Then you run the new program using the new text/xml scripts, and store
the results.  These results are now the new "standard"/expected results
by which all subsequent tests can be judged.

This doesn't work with for new test, which we create but should get us
over the hurdle of creating the initial/current results.
</Sean>

> > 
> <david>
> Here are some things I think are important or a good idea.
> 
> 1. separate logical content of tests and results from 
> formatting details.
> 2. Separate platform/execution context from content of test.
> 3. Concentrate on testing drivers used by application programs.
> 4. Have as much of the testing framework work on all 
> platforms as possible
> with as little modification as possible.
> 5. File based version control.
> 6. test suite updates available in small pieces. ( not the 
> whole tcs db)
> 
> I hope you can comment on my samples posted in a different message.
> Thanks!
> </david>

<Sean>
Don't think I fully understand what you mean in #2.

WRT #3, I would suggest that the efforts should be concentrated on
testing the engine and _then_ testing the data drivers.
</Sean>

Re: [Firebird-test] Catching up on postings

[reed m. <rf...@cr...>]

"Leyne, Sean" wrote:
> 
[...]
> Ah, there's your problem!  You're using InterClient! <grin>
> 
> The only problem I have with your points above is that the initial focus
> is on testing the engine functionality.
> 
> Using any data access layer (jdbc, odbc, IBO...) introduces into the
> testing scheme possible 'artifacts'/errors related to the data access
> layer and not engine functionality.
> 
Mot only that, it makes it impossible to test functionality which
is not exposed by your particular access layer(s).

> The current ISQL approach virtually eliminated these type of errors,
> saving testers from hunting down these 'false positive' results.
> Furthermore, when it comes to diagnosing anomolies in data results, ISQL
> has been used as the "gold standard" for determining whether a problem
> is engine or data access layer related.
> 
Emm, actually, embedded GDML (i.e. GPRE) is the gold standard.
isql only lets you test DSQL. 

[...]
--
Reed Mideke
email: rfm(at)cruzers.com  -If that fails:  rfm(at)portalofevil.com

Re: [Firebird-test] Catching up on postings

[David J. <dav...@ea...>]

Hi,
On 2001.05.15 15:57:47 -0400 reed mideke wrote:
> "Leyne, Sean" wrote:
> > 
> [...]
> > Ah, there's your problem!  You're using InterClient! <grin>
> > 
> > The only problem I have with your points above is that the initial
> focus
> > is on testing the engine functionality.
> > 
> > Using any data access layer (jdbc, odbc, IBO...) introduces into the
> > testing scheme possible 'artifacts'/errors related to the data access
> > layer and not engine functionality.
> > 
> Mot only that, it makes it impossible to test functionality which
> is not exposed by your particular access layer(s).

obviously true.
> 
> > The current ISQL approach virtually eliminated these type of errors,
> > saving testers from hunting down these 'false positive' results.
> > Furthermore, when it comes to diagnosing anomolies in data results,
> ISQL
> > has been used as the "gold standard" for determining whether a problem
> > is engine or data access layer related.
> > 
> Emm, actually, embedded GDML (i.e. GPRE) is the gold standard.
> isql only lets you test DSQL. 
> 

Yes, I agree GDML is the gold standard.  However, I suspect almost noone is
going to start using Firebird to use either GDML or embedded sql.  99% of
new users (I'm making this numer up, but do you disagree?) are going to use
ejbs, delphi + ibo, odbc, or plain jdbc.  So if the tests available for
these drivers pass, this will make almost everyone happy.  If they fail,
true it is harder to find the problem than if fewer layers are present. 
However, these are not unit tests, they are functional tests. If there is
an error, lets write a unit test on the broken part when we fix it.

As to the 'limited functionality' question, as far as I know the major
missing piece in the jdbc category is reasonable transaction support.  I
think this will be addressed by the new java jca driver.  I think multiple
database support can be addressed by using the catalog and possibly schema
fields in the metadata.  I would like to know what other important pieces
of functionality are missing from a full jdbc/jca driver.

I have also suggested that we might be able to write a simple tool that
took the sql statement, generated a c embedded sql program, compiled it,
got the results, and compared them to the standard. 
I think it is really important to keep the logical content of what we are
testing (sql + logical structure and content of resuts) separated from the
particular testing driver we are using.

david jencks

> [...]
> --
> Reed Mideke
> email: rfm(at)cruzers.com  -If that fails:  rfm(at)portalofevil.com
> 
> _______________________________________________
> Firebird-test mailing list
> Fir...@li...
> http://lists.sourceforge.net/lists/listinfo/firebird-test
>