firebird-test — Firebird engine testing, test flight/suite discussion/information

Re: [Firebird-test] Exported TCS tests???

[David J. <dav...@ea...>]

Hi,
I did this here on my computer, then I started extracting the actual tests
to put into my xml based testing/scripting framework, although that project
didn't get too far.  Right now I have mislaid the output xml.  Would you
like the output or the program to generate it (requires java, and a good
java xml parser -- crimsom or xerxes)?  If you want the xml, where should I
put it - I think it is several mb, maybe too large to email.

david jencks

On 2001.06.16 20:45:52 -0400 "Leyne, Sean" wrote:
> All,
> 
> I thought that I saw, somewhere, where someone had exported the TCS
> Suite test scripts to text (perhaps XML), but I can't seem to find them.
> 
> Can someone please point me in the right direction.
> 	
> Thanks in advance.
> 
> 
> Sean
> 
> _______________________________________________
> Firebird-test mailing list
> Fir...@li...
> http://lists.sourceforge.net/lists/listinfo/firebird-test
> 

[Firebird-test] Re: [Firebird-devel] Exported TCS tests???

[Neil M. <nm...@zi...>]

On Sat, Jun 16, 2001 at 08:45:52PM -0400, Leyne, Sean wrote:
> All,
> 
> I thought that I saw, somewhere, where someone had exported the TCS
> Suite test scripts to text (perhaps XML), but I can't seem to find them.
> 
> Can someone please point me in the right direction.
> 	

There are some perl scripts by Frank in the main CVS in firebird/fsg/TCS 
which export the contents of gtcs.gdb to test files and import them
again. The comment in the howto.txt says the resultant database not work
but something has been fixed since as they do.

To add a test for by group by udf change I made, I exported the tests
and tables, edited the text files and imported again. This database has
been checked back in to the TCS cvs tree. 


-- 
Neil McCalden                @home     nm...@zi...     

[Firebird-test] Exported TCS tests???

[Leyne, S. <sl...@at...>]

All,

I thought that I saw, somewhere, where someone had exported the TCS
Suite test scripts to text (perhaps XML), but I can't seem to find them.

Can someone please point me in the right direction.
	
Thanks in advance.


Sean

[Firebird-test] Very basic stress tester

[John B. <bel...@cs...>]

Hi all,
	I just committed in my sandbox (interbase/firebird/bellardo) a VERY 
basic stress testing program.  It is written using DSQL and *nix 
commands.  If nothing else it gives us something to start with.  It 
performs a join on 2 tables in the msg.gdb database from a number of 
connections at the same time (or as close as possible).  There is a 
makefile (works with Darwin only, but easy to change).  Run the program 
with no parameters for usage.  Let me know what you think, and lets get 
1.0 out the door....

-John

Re: [Firebird-test] Firebird Testing - A plan for discussion

[Maverick <ye...@ho...>]

I agree with Martijn.  I am offering some time to test FB.  However, I did
not think that I would need to install a development environment for FB
itself.

I was thinking more along the lines of preparing a set of scripts to run
against my database and comparing the consistency of results and
performance.  My db is about 50MB in size, soon to grow to about 300MB, used
in a data-warehouse like environment (I know, it's not ideal, but the
performance is ok with aggregations done upfront).

My environment is Win2000/WinME/Peanut Linux 8.4 (small but enough).

--
Regards
Ray Mond

""Martijn Tonies"" <m.t...@up...> wrote in message
news:005e01c0dcc1$1bae9900$0100a8c0@seal...
> Hi all,
>
> I offered testing some time ago and already had a few email with Mark and
> Tord.
>
> One of the points I very much dislike (and actually restrain me from
> testing) is the need to install a (visual) C++ suite. This is a big no-no
> for me... Besides the fact that I have zero C++ (alright, just a little)
> knowledge, I do not want to install a complete development suite for
testing
> IB/Firebird.
>
> I would like to see a set of scripts/application that runs a few
> tests/durability tests/volume tests etc...
>
> In general, I think that (mind you, I don't have knowledge about TCS):
>
> - after every change, the programmer should think of what has changed
> opposed to the code before the change
> - if it was changed because of a bug:
>   1) is the normal behaviour still available
>   2) does the original 'buggy behaviour' still exists (probably not :))
>   3) do the tests that test extraordinary situations still run
> - new functionality:
>   1) the programmer should make a detailed description of how this should
> work
>   2) make a test plan, with extraordinary situations, the normal behaviour
> and possible buggy situations when changed (in the form: when I do this,
the
> output should be: <something>)
>   3) test everything before declaring it 'working' code
>
> Furthermore I think that:
> 1) nothing should be installed (except the IB/FB client)
> 2) there should be detailed instructions for volume testing as well as a
> filling procedure
> 3) there should be detailed instructions considering the output of the
tests
> 4) there should be an easy-to-use application/set of scripts that test all
> or parts of the engine
> 5) there should be ways that the whole testing process can be devided into
> parts that can be performed by several people (or groups of people)
instead
> of a single testbed for all tests
> 6) there should be applications for durability and load testing
>
> Well, that's about it.
>
> For starters, I can contribute with testing on Win2000, Linux (Suse 7.1)
> (both AMD CPUs).
>
>
> Comments anyone?
>
> Martijn Tonies
> InterBase Workbench - the developer tool for InterBase and Firebird
> http://www.interbaseworkbench.com
>
> Upscene Productions
> http://www.upscene.com
>
> "This is an object-oriented system.
> If we change anything, the users object."
>
>
>
>
>
> _______________________________________________
> Firebird-test mailing list
> Fir...@li...
> http://lists.sourceforge.net/lists/listinfo/firebird-test
>

Re: [Firebird-test] Catching up on postings

[David J. <dav...@ea...>]

Hi,
On 2001.05.15 15:57:47 -0400 reed mideke wrote:
> "Leyne, Sean" wrote:
> > 
> [...]
> > Ah, there's your problem!  You're using InterClient! <grin>
> > 
> > The only problem I have with your points above is that the initial
> focus
> > is on testing the engine functionality.
> > 
> > Using any data access layer (jdbc, odbc, IBO...) introduces into the
> > testing scheme possible 'artifacts'/errors related to the data access
> > layer and not engine functionality.
> > 
> Mot only that, it makes it impossible to test functionality which
> is not exposed by your particular access layer(s).

obviously true.
> 
> > The current ISQL approach virtually eliminated these type of errors,
> > saving testers from hunting down these 'false positive' results.
> > Furthermore, when it comes to diagnosing anomolies in data results,
> ISQL
> > has been used as the "gold standard" for determining whether a problem
> > is engine or data access layer related.
> > 
> Emm, actually, embedded GDML (i.e. GPRE) is the gold standard.
> isql only lets you test DSQL. 
> 

Yes, I agree GDML is the gold standard.  However, I suspect almost noone is
going to start using Firebird to use either GDML or embedded sql.  99% of
new users (I'm making this numer up, but do you disagree?) are going to use
ejbs, delphi + ibo, odbc, or plain jdbc.  So if the tests available for
these drivers pass, this will make almost everyone happy.  If they fail,
true it is harder to find the problem than if fewer layers are present. 
However, these are not unit tests, they are functional tests. If there is
an error, lets write a unit test on the broken part when we fix it.

As to the 'limited functionality' question, as far as I know the major
missing piece in the jdbc category is reasonable transaction support.  I
think this will be addressed by the new java jca driver.  I think multiple
database support can be addressed by using the catalog and possibly schema
fields in the metadata.  I would like to know what other important pieces
of functionality are missing from a full jdbc/jca driver.

I have also suggested that we might be able to write a simple tool that
took the sql statement, generated a c embedded sql program, compiled it,
got the results, and compared them to the standard. 
I think it is really important to keep the logical content of what we are
testing (sql + logical structure and content of resuts) separated from the
particular testing driver we are using.

david jencks

> [...]
> --
> Reed Mideke
> email: rfm(at)cruzers.com  -If that fails:  rfm(at)portalofevil.com
> 
> _______________________________________________
> Firebird-test mailing list
> Fir...@li...
> http://lists.sourceforge.net/lists/listinfo/firebird-test
> 

Re: [Firebird-test] Catching up on postings

[reed m. <rf...@cr...>]

"Leyne, Sean" wrote:
> 
[...]
> Ah, there's your problem!  You're using InterClient! <grin>
> 
> The only problem I have with your points above is that the initial focus
> is on testing the engine functionality.
> 
> Using any data access layer (jdbc, odbc, IBO...) introduces into the
> testing scheme possible 'artifacts'/errors related to the data access
> layer and not engine functionality.
> 
Mot only that, it makes it impossible to test functionality which
is not exposed by your particular access layer(s).

> The current ISQL approach virtually eliminated these type of errors,
> saving testers from hunting down these 'false positive' results.
> Furthermore, when it comes to diagnosing anomolies in data results, ISQL
> has been used as the "gold standard" for determining whether a problem
> is engine or data access layer related.
> 
Emm, actually, embedded GDML (i.e. GPRE) is the gold standard.
isql only lets you test DSQL. 

[...]
--
Reed Mideke
email: rfm(at)cruzers.com  -If that fails:  rfm(at)portalofevil.com

RE: [Firebird-test] Catching up on postings

[Leyne, S. <sl...@at...>]

David,

See my comments below:

> -----Original Message-----
> From: David Jencks [mailto:dav...@ea...]
> Sent: Tuesday, May 15, 2001 12:25 AM
> To: fir...@li...
> Subject: RE: [Firebird-test] Catching up on postings
> 
> > <Sean>
> > That's great news! 
> > 
> > I'm wondering though, why did you choose XML as the file format, as
> > compared to straight text?
> > </Sean>
> > 
> <david>
> XML structure/parsers give you the ability to find just what 
> you want in an 
> arbitrarily large document without having to write a parser.  
> Navigation is
> also plausible.  It should be very easy to provide a nice 
> html views of
> resultsets from the xml translation.  The xml structure makes 
> it easy to
> pinpoint where differences between the old and new results 
> are... give a
> path to  the elements.  A big reason though is probably because this
> started life as an ant task, and ant build files are xml based.
> </david>

<Sean>
David, are you proposing to store the test results in a separate file
from the original script or in the same file?

Personally, I think that the results should be stored separately.

I can appreciate the benefits of using XML for storing the results, it
would allow for a quick analysis of the elements to determine
discrepancies.

I must admit that I really prefer the simplicity of straight text for
the testing scripts themselves, perhaps I need to see a simple example
to appreciate the benefits of the structured layout.
</Sean>

> > 
> > I don't think reliance on isql is appropriate.
> > 
> > <Sean>
> > Before Ann has a chance to ask, why not?  
> > 
> > ISQL needs to be tested, so why not use it as part of the tests?
> > </Sean>
> 
> <david>I suppose you are right that it needs testing.  I have 
> not had good
> results with isql, maybe I don't know how to make it commit.  
> It doesn't
> usually show me what's in the database, put there by 
> interclient.  Also as
> far as I know you can't test parameterized queries with it. Testing
> threading/parallel execution is as far as I know impossible. 
> It seems to me
> that testing the ways client programs will use the db is more 
> important
> than testing a scripting engine.  So for my money, the most important
> testing is through jdbc drivers, odbc drivers, ibo, ibx, bde, 
> c code that
> uses the dynamic sql interface, static sql, and gdml, in 
> approximately that
> order (first 3 can be shuffled).</david>
> > 

<Sean>

Ah, there's your problem!  You're using InterClient! <grin>

The only problem I have with your points above is that the initial focus
is on testing the engine functionality.  

Using any data access layer (jdbc, odbc, IBO...) introduces into the
testing scheme possible 'artifacts'/errors related to the data access
layer and not engine functionality.  

The current ISQL approach virtually eliminated these type of errors,
saving testers from hunting down these 'false positive' results.
Furthermore, when it comes to diagnosing anomolies in data results, ISQL
has been used as the "gold standard" for determining whether a problem
is engine or data access layer related.

Don't get me wrong, I agree that data access layer testing is necessary
and should be considered as part of the overall goal.
</Sean>

> > <Sean>
> > Don't understand what you mean about extracting the SQL?
> > </Sean>
> <david>
> Most of the TCS I looked at(not very much) consisted of a simple sql
> statement plus a c program to run it and produce text output: 
> elsewhere in
> TCS is the expected output.  The format of the output is 
> highly dependent
> on the actual c program used.  To me, the important parts are 
> in order (1)
> the SQL (2) the logical structure and contents of the results (3) the
> logical setup information (4) the c program (5) the text layout of the
> results.
> 
> Finding the SQL and putting it into the xml format I am 
> proposing is fairly
> easy.  Finding the expected results is a little harder.  I 
> don't know how
> to automatically translate the expected results into the xml 
> result set
> format I propose.  I think someone has to look at it and decide if the
> meaning is the same.  I suppose it might be possible to take 
> the xml and
> transform it into text of the same format as TCS, and compare 
> that...I'll
> have to think about this some more.
> </david>

<Sean>
David, it was my understanding that the suite involved sending the
script commands to ISQL and then capturing the generated results.  I
wasn't aware of the associated c programs, so perhaps I (and others)
need an education about the functionality/implementation of the TCS
suite and the role of the C programs.

Accordingly, I _had_ been thinking it was actually fairly
straightforward to capture the results (although this may need some more
thought).

First, you successfully run the existing TCS suite on an existing engine
release (0.9-4).  This ensures that the engine is producing the expected
results.

Then you run the new program using the new text/xml scripts, and store
the results.  These results are now the new "standard"/expected results
by which all subsequent tests can be judged.

This doesn't work with for new test, which we create but should get us
over the hurdle of creating the initial/current results.
</Sean>

> > 
> <david>
> Here are some things I think are important or a good idea.
> 
> 1. separate logical content of tests and results from 
> formatting details.
> 2. Separate platform/execution context from content of test.
> 3. Concentrate on testing drivers used by application programs.
> 4. Have as much of the testing framework work on all 
> platforms as possible
> with as little modification as possible.
> 5. File based version control.
> 6. test suite updates available in small pieces. ( not the 
> whole tcs db)
> 
> I hope you can comment on my samples posted in a different message.
> Thanks!
> </david>

<Sean>
Don't think I fully understand what you mean in #2.

WRT #3, I would suggest that the efforts should be concentrated on
testing the engine and _then_ testing the data drivers.
</Sean>

RE: [Firebird-test] Catching up on postings

[<pc...@at...>]

Hi,

On 15 May 2001, at 0:25, David Jencks wrote:

> Here are some things I think are important or a good idea.
> 
> 1. separate logical content of tests and results from formatting details.
> 2. Separate platform/execution context from content of test.
> 3. Concentrate on testing drivers used by application programs.
> 4. Have as much of the testing framework work on all platforms as possible
> with as little modification as possible.
> 5. File based version control.
> 6. test suite updates available in small pieces. ( not the whole tcs db)

I think that this is the best way to go. We would have test 
definitions (with data, executed/tested functionality and expected 
results) in portable, easy to manipulate format (XML). When test 
framework would be defined, it could be easily implemented in 
many languages, with different access methods and on any 
platform. 

Just a side note: We should separate the test database content 
XML from tests description XML (and reference it).

Actually, I *love* this idea more and more.

Best regards
-- Pavel

RE: [Firebird-test] Catching up on postings

[David J. <dav...@ea...>]

Hi,

First, I want to apologize for the not very pleasant tone of my last
message--sorry.

Here are a few thoughts interspersed
On 2001.05.14 09:49:47 -0400 "Leyne, Sean" wrote:
> David,
> 
> -----Original Message-----
> From: 	David Jencks [mailto:dav...@ea...] 
> Sent:	Monday, May 14, 2001 1:26 AM
> To:	fir...@li...
> Subject:	Re: [Firebird-test] Catching up on postings
> 
> Hi,
> 
> Well, yes, I already started talking about it.
> 
> <Sean>
> I didn't mean to imply that you hadn't.  I just wanted to re-focus the
> discussion to a higher level before we got immersed in the almost
> inevitable "religious" wars regarding C++ vs. Java vs. Python...
> 
> "Talking about the tools before we have defined the problem is like
> putting the cart before the horse."
> </Sean>
<david>
Well, perhaps, but there is also the xprogramming idea of implementing the
simplest possible functionality for the most important use, then adding
more for the next most important use while refactoring to improve the
design,....  The first version of what I am working on I wrote to (1) have
a java based scripting method and (2) test the interclient changes I made a
while back.  I'm now rewriting it to be independent of ant and provide
easier to use regression tests.
</david>
> 
> 
> I have nearly completed a java scripting/testing solution based on xml
> script/test files.  It can include the results of db access in the xml
> file, and compare the current results with a stored reference.  Being
> written in java, it will be easy to extend to test parallel db access.
> 
> <Sean>
> That's great news! 
> 
> I'm wondering though, why did you choose XML as the file format, as
> compared to straight text?
> </Sean>
> 
<david>
XML structure/parsers give you the ability to find just what you want in an
arbitrarily large document without having to write a parser.  Navigation is
also plausible.  It should be very easy to provide a nice html views of
resultsets from the xml translation.  The xml structure makes it easy to
pinpoint where differences between the old and new results are... give a
path to  the elements.  A big reason though is probably because this
started life as an ant task, and ant build files are xml based.
</david>
>  
> Obviously it is best adapted to testing via jdbc drivers, 
> 
> <Sean>
> Have you looked at the Sun JDBC compliance testing routines?
> </Sean>
> 
<david> Only marginally... they mostly test the jdbc driver rather than db
functionality as far as I know</david>
> but I think it might be worthwile investigating how hard it would be to
> write adapters to
> use odbc drivers, IBO, and possibly generating dynamic and static sql c
> or
> c++ programs.  
> 
> <Sean>
> Personally, I'd like to leave the issue of IBO and dynamic/static C or
> C++ programs to a much, much later time.  We have enough on our plate.
> </Sean>
> 
<david> It did occur to me that at least some c client functionality might
be testable with exactly the same code by using the much reviled jdbc-odbc
bridge and Jim's odbc driver.</david>
> I think having a single cross platform execution engine would be rather
> desirable.  
> 
> <Sean>
> Agreed!
> </Sean>
> 
> I don't think reliance on isql is appropriate.
> 
> <Sean>
> Before Ann has a chance to ask, why not?  
> 
> ISQL needs to be tested, so why not use it as part of the tests?
> </Sean>

<david>I suppose you are right that it needs testing.  I have not had good
results with isql, maybe I don't know how to make it commit.  It doesn't
usually show me what's in the database, put there by interclient.  Also as
far as I know you can't test parameterized queries with it. Testing
threading/parallel execution is as far as I know impossible. It seems to me
that testing the ways client programs will use the db is more important
than testing a scripting engine.  So for my money, the most important
testing is through jdbc drivers, odbc drivers, ibo, ibx, bde, c code that
uses the dynamic sql interface, static sql, and gdml, in approximately that
order (first 3 can be shuffled).</david>
> 
> 
> 
> Translating the TCS tests to any other format is going to be a large
> amount
> of work.  Its easy enough to dump say each test into an xml file, but
> extracting the sql and expected results into a more reasonable format
> will
> be time consuming.  The NIST tests look only slightly easier to move to
> a
> different system than TCS.
> 
> <Sean>
> Don't understand what you mean about extracting the SQL?
> </Sean>
<david>
Most of the TCS I looked at(not very much) consisted of a simple sql
statement plus a c program to run it and produce text output: elsewhere in
TCS is the expected output.  The format of the output is highly dependent
on the actual c program used.  To me, the important parts are in order (1)
the SQL (2) the logical structure and contents of the results (3) the
logical setup information (4) the c program (5) the text layout of the
results.

Finding the SQL and putting it into the xml format I am proposing is fairly
easy.  Finding the expected results is a little harder.  I don't know how
to automatically translate the expected results into the xml result set
format I propose.  I think someone has to look at it and decide if the
meaning is the same.  I suppose it might be possible to take the xml and
transform it into text of the same format as TCS, and compare that...I'll
have to think about this some more.
</david>
> 
> Would a provisional dtd for what I am working on and sample document be
> helpful? Both are subject to change by me before I show the code, but
> should give an idea of what I'm after.
> 
> <Sean>
> Personally, I'd like to talk about general approach before diving into
> the details.
> </Sean>
> 
<david>
Here are some things I think are important or a good idea.

1. separate logical content of tests and results from formatting details.
2. Separate platform/execution context from content of test.
3. Concentrate on testing drivers used by application programs.
4. Have as much of the testing framework work on all platforms as possible
with as little modification as possible.
5. File based version control.
6. test suite updates available in small pieces. ( not the whole tcs db)

I hope you can comment on my samples posted in a different message.
Thanks!
</david>


> 
> _______________________________________________
> Firebird-test mailing list
> Fir...@li...
> http://lists.sourceforge.net/lists/listinfo/firebird-test
> 

Re: [Firebird-test] Firebird Testing - A plan for discussion

[Martijn T. <m.t...@up...>]

Hi all,

I offered testing some time ago and already had a few email with Mark and
Tord.

One of the points I very much dislike (and actually restrain me from
testing) is the need to install a (visual) C++ suite. This is a big no-no
for me... Besides the fact that I have zero C++ (alright, just a little)
knowledge, I do not want to install a complete development suite for testing
IB/Firebird.

I would like to see a set of scripts/application that runs a few
tests/durability tests/volume tests etc...

In general, I think that (mind you, I don't have knowledge about TCS):

- after every change, the programmer should think of what has changed
opposed to the code before the change
- if it was changed because of a bug:
  1) is the normal behaviour still available
  2) does the original 'buggy behaviour' still exists (probably not :))
  3) do the tests that test extraordinary situations still run
- new functionality:
  1) the programmer should make a detailed description of how this should
work
  2) make a test plan, with extraordinary situations, the normal behaviour
and possible buggy situations when changed (in the form: when I do this, the
output should be: <something>)
  3) test everything before declaring it 'working' code

Furthermore I think that:
1) nothing should be installed (except the IB/FB client)
2) there should be detailed instructions for volume testing as well as a
filling procedure
3) there should be detailed instructions considering the output of the tests
4) there should be an easy-to-use application/set of scripts that test all
or parts of the engine
5) there should be ways that the whole testing process can be devided into
parts that can be performed by several people (or groups of people) instead
of a single testbed for all tests
6) there should be applications for durability and load testing

Well, that's about it.

For starters, I can contribute with testing on Win2000, Linux (Suse 7.1)
(both AMD CPUs).


Comments anyone?

Martijn Tonies
InterBase Workbench - the developer tool for InterBase and Firebird
http://www.interbaseworkbench.com

Upscene Productions
http://www.upscene.com

"This is an object-oriented system.
If we change anything, the users object."

[Firebird-test] Example of xml format tests

[David J. <dav...@ea...>]

Hi,

I am attaching a simple example of the xml testing format I am thinking
about, with the dtd.  This has 3 tests from TCS in it.  The xml file
structure is kind of like this:

target
  transaction
    statement
      sql
      resultset (from statement, for comparison with future runs)
        row-spec(metadata)
          column-spec*
        rows(the data)
          row*
            column*
      diffs
        diff*(path to reference and comparison elements, and message about
difference)


I messed with some reference data by hand to generate a diff.  

Obvious future enhancements (in java) include:
using xpath for diff paths to elements, to enable tools to find it
automatically.
Extending target-ref to refer to other files.
Parallel execution.
Expanding databasemetadata support.(currently getTables and getColumns are
supported)

Problems include handling binary blobs, such as are found in the system
tables. They generate characters illegal in xml.

Perhaps these can be base64 encoded?

Does anyone have an idea how hard it would be to make a fake jdbc driver
wrapper for e.g. ibo? (just implementing statement and simple transaction
stuff should be ok.)

Anyone have experience using jdbc-odbc bridge? Maybe this would work with
Jim's odbc driver to get tests of c platform functionality.

I will try to get the code for this to my website soon.

Hoping for comments
Thanks
David Jencks





    

Re: [Firebird-test] Catching up on postings

[Frank Schlottmann-G. <sch...@t-...>]

Ann W. Harrison wrote:

> At 12:25 AM 5/14/2001 -0400, Leyne, Sean wrote:
> 
> 
> >Any other suggestions?
> 
> MySQL has some tests that they use to prove that no one
> is more SQL than mySQL.  We should probably have a look at
> them.

I remember that I tried it. It's no fun. Segfaulting all the way. This is a 
job for someone with a deeper knowledge of c and SQL than me.
(or more time to get there)
BTW they don't supply a default interbase testbed  (Honi soit qui mal y 
pense) :-)

Frank
-- 
"Fascinating creatures, phoenixes. They can carry immensely heavy loads,
 their tears have healing powers and they make highly faithful pets."
         - J.K. Rowling
http://firebird.sourceforge.net

Re: [Firebird-test] Catching up on postings

[Ann W. H. <aha...@ib...>]

At 12:25 AM 5/14/2001 -0400, Leyne, Sean wrote:


>Any other suggestions?

MySQL has some tests that they use to prove that no one
is more SQL than mySQL.  We should probably have a look at
them.


Regards,

Ann

RE: [Firebird-test] Catching up on postings

[Leyne, S. <sl...@at...>]

-----Original Message-----
From: 	Jason Chapman (JAC2) [mailto:ja...@ja...] 
Sent:	Monday, May 14, 2001 6:28 AM
To:	fir...@li...
Subject:	Re: [Firebird-test] Catching up on postings


<Snip>

> 3) All of the testing programs (including any script editors)
> should portable, to run on any platform which the engine runs on.
This may hamper development, I could work on some bits in Delphi for
Win32,
but that would under the "must be under all OS'" be a no-no.

<Sean>
Don't get me wrong, I'm one of Bill Gates drones, I know practically
nothing about C, C++ or *nux so I'd love to say that we should write the
testing programs in Delphi/Kylix.  However, the frank reality is that
the testing programs need to apply much more broadly.
</Sean>

> 4) Any choice of development tools (Delphi, C++, Python, Java or MS
> C# <big grin>) should carefully consider the overall expertise of the
> project members.
3 damages 4. Suddenly we've lost Delphi + MS anything.

<Sean>
Keep in mind that the testing programs are really designed (even TCS
today) as a command processor; taking in commands (scripts), executing
the commands and then reporting the results.  So, in the long run most
of the work in the testing process will be in the management/creation of
testing scenarios/scripts, which is something that everybody can make
very important contributions.

The testing programs themselves will probably require little maintenance
of their own, so the fact that most of us would really feel more
comfortable using Delphi which isn't cross-platform should really matter
in the long run - there will always be someone who can work on the
programs for us <grin>.

There, however, could be a role for Delphi in creating a nifty script
editor/manager, to save use from the depths/details of the XML (to use
David's example).
</Sean>

RE: [Firebird-test] Catching up on postings

[Leyne, S. <sl...@at...>]

David,

-----Original Message-----
From: 	David Jencks [mailto:dav...@ea...] 
Sent:	Monday, May 14, 2001 1:26 AM
To:	fir...@li...
Subject:	Re: [Firebird-test] Catching up on postings

Hi,

Well, yes, I already started talking about it.

<Sean>
I didn't mean to imply that you hadn't.  I just wanted to re-focus the
discussion to a higher level before we got immersed in the almost
inevitable "religious" wars regarding C++ vs. Java vs. Python...

"Talking about the tools before we have defined the problem is like
putting the cart before the horse."
</Sean>


I have nearly completed a java scripting/testing solution based on xml
script/test files.  It can include the results of db access in the xml
file, and compare the current results with a stored reference.  Being
written in java, it will be easy to extend to test parallel db access.

<Sean>
That's great news! 

I'm wondering though, why did you choose XML as the file format, as
compared to straight text?
</Sean>

 
Obviously it is best adapted to testing via jdbc drivers, 

<Sean>
Have you looked at the Sun JDBC compliance testing routines?
</Sean>

but I think it might be worthwile investigating how hard it would be to
write adapters to
use odbc drivers, IBO, and possibly generating dynamic and static sql c
or
c++ programs.  

<Sean>
Personally, I'd like to leave the issue of IBO and dynamic/static C or
C++ programs to a much, much later time.  We have enough on our plate.
</Sean>

I think having a single cross platform execution engine would be rather
desirable.  

<Sean>
Agreed!
</Sean>

I don't think reliance on isql is appropriate.

<Sean>
Before Ann has a chance to ask, why not?  

ISQL needs to be tested, so why not use it as part of the tests?
</Sean>



Translating the TCS tests to any other format is going to be a large
amount
of work.  Its easy enough to dump say each test into an xml file, but
extracting the sql and expected results into a more reasonable format
will
be time consuming.  The NIST tests look only slightly easier to move to
a
different system than TCS.

<Sean>
Don't understand what you mean about extracting the SQL?
</Sean>

Would a provisional dtd for what I am working on and sample document be
helpful? Both are subject to change by me before I show the code, but
should give an idea of what I'm after.

<Sean>
Personally, I'd like to talk about general approach before diving into
the details.
</Sean>

Re: [Firebird-test] FBT - firebird tests

[Luis F. <lou...@no...>]

Hello,

In developing the test scripts does it matter what environment is used.
I was thinking perhaps of a Delphi/Kylix app to serve as  a framework. 
Of course, that leaves out Solaris and others. Whatever the front end
app is,  I think that by keeping the tests in a format that is easy to
get to and use, several different tools could be developed by users for
either Open Source or internal testing.  I have two extra machines that
I plan on loading Linux on for testing interbase and have not had the
time to figure TCS out. I would be glad to get involved in the testing
effort.

- Lou

Tord Hammer wrote:
> 
> Hi,
> 
> up to now the only way to test the changes in firebird
> is the old (summer 2000) test suite from borland (aka TCS).
> 
> I suggest a new test systems called FBT ;-) which works
> like the PostgresQL one. It would be strict script-based,
> which means all tests and expected results are stored
> in text files. Groups of tests (calles series in TCS) and
> group of series (calles meta-series) would be text files
> too.
>

Re: [Firebird-test] Catching up on postings

[Jason C. (JAC2) <ja...@ja...>]

""Leyne, Sean"" <sl...@at...> wrote in message
news:D3AC71AF0C0BD411877E0000E20DEAD0180428@MDA_NT...
> All,
>
> First, please allow me to apologize for not reading the postings sent
> over the last couple of days prior to sending my "plan for discussion"
> posting.  I wasn't used to much activity in this list and didn't even
> think to check.
>
> While I am impressed with the discussions that have taken place, I think
> we need to establish the goals which need to be address by the testing
> processes.
>
> I see the goals as:
>
> 1) Users running the testing programs should be insulated from the
> actual implementation/coding of the programs.  They should be able to
> download a ZIP/TAR with contains an executable, run it, review the
> results and perhaps create new scripts.
>
Fully Agree.  They should possibly have release notes / change log just for
interest after the tests.

> 2) The execution of the testing program should involve a minimal
> installation footprint; the db client is expected, installing a C++
> compiler should not be.
I don't mind if we have to install a C++ compiler as long as there are idiot
proof instructions.

> 3) All of the testing programs (including any script editors)
> should portable, to run on any platform which the engine runs on.
This may hamper development, I could work on some bits in Delphi for Win32,
but that would under the "must be under all OS'" be a no-no.

> 4) Any choice of development tools (Delphi, C++, Python, Java or MS
> C# <big grin>) should carefully consider the overall expertise of the
> project members.
3 damages 4. Suddenly we've lost Delphi + MS anything.

JAC.

Re: [Firebird-test] Catching up on postings

[David J. <dav...@ea...>]

Hi,

Well, yes, I already started talking about it.

I have nearly completed a java scripting/testing solution based on xml
script/test files.  It can include the results of db access in the xml
file, and compare the current results with a stored reference.  Being
written in java, it will be easy to extend to test parallel db access. 
Obviously it is best adapted to testing via jdbc drivers, but I think it
might be worthwile investigating how hard it would be to write adapters to
use odbc drivers, IBO, and possibly generating dynamic and static sql c or
c++ programs.  I think having a single cross platform execution engine
would be rather desirable.  I don't know how to do this except in java.  I
don't think reliance on isql is appropriate.

Translating the TCS tests to any other format is going to be a large amount
of work.  Its easy enough to dump say each test into an xml file, but
extracting the sql and expected results into a more reasonable format will
be time consuming.  The NIST tests look only slightly easier to move to a
different system than TCS.

Would a provisional dtd for what I am working on and sample document be
helpful? Both are subject to change by me before I show the code, but
should give an idea of what I'm after.

thanks
David Jencks



On 2001.05.14 00:25:27 -0400 "Leyne, Sean" wrote:
> All,
> 
> First, please allow me to apologize for not reading the postings sent
> over the last couple of days prior to sending my "plan for discussion"
> posting.  I wasn't used to much activity in this list and didn't even
> think to check.
> 
> While I am impressed with the discussions that have taken place, I think
> we need to establish the goals which need to be address by the testing
> processes.
> 
> I see the goals as:
> 
> 1)	Users running the testing programs should be insulated from the
> actual implementation/coding of the programs.  They should be able to
> download a ZIP/TAR with contains an executable, run it, review the
> results and perhaps create new scripts.
> 
> 2)	The execution of the testing program should involve a minimal
> installation footprint; the db client is expected, installing a C++
> compiler should not be.
> 
> 3)	All of the testing programs (including any script editors)
> should portable, to run on any platform which the engine runs on.
> 
> 4)	Any choice of development tools (Delphi, C++, Python, Java or
> MS
> C# <big grin>) should carefully consider the overall expertise of the
> project members.  
> 
> Other goals?
> 
> 
> Next, in order to decide how to proceed, we need to understand what the
> scope of the TCS suite functionality.  I, personally, don't understand
> why the current suite required such a elaborate shell/scripting
> environment and how that fit's in to the script which are executed.
> Perhaps someone could write up a explanation for all the member of the
> list -- so that we all understand who the piece are currently
> constructed.  Any volunteers?
> 
> 
> Finally, I would suggest that before we go writing any code that we need
> to explore the options available to us.  The options which have been
> mentioned so far include:
> 
> - modifying the current TCS suite
> - PostgreSQL testing suite
> - NIST suite
> - XP test framework
> - Open Source Database Benchmark (http://freshmeat.net/projects/osdb/)
> - CppUnit 
> 
> Any other suggestions?
> 
> 
> Sean
> 
> _______________________________________________
> Firebird-test mailing list
> Fir...@li...
> http://lists.sourceforge.net/lists/listinfo/firebird-test
> 

[Firebird-test] Catching up on postings

[Leyne, S. <sl...@at...>]

All,

First, please allow me to apologize for not reading the postings sent
over the last couple of days prior to sending my "plan for discussion"
posting.  I wasn't used to much activity in this list and didn't even
think to check.

While I am impressed with the discussions that have taken place, I think
we need to establish the goals which need to be address by the testing
processes.

I see the goals as:

1)	Users running the testing programs should be insulated from the
actual implementation/coding of the programs.  They should be able to
download a ZIP/TAR with contains an executable, run it, review the
results and perhaps create new scripts.

2)	The execution of the testing program should involve a minimal
installation footprint; the db client is expected, installing a C++
compiler should not be.

3)	All of the testing programs (including any script editors)
should portable, to run on any platform which the engine runs on.

4)	Any choice of development tools (Delphi, C++, Python, Java or MS
C# <big grin>) should carefully consider the overall expertise of the
project members.  

Other goals?


Next, in order to decide how to proceed, we need to understand what the
scope of the TCS suite functionality.  I, personally, don't understand
why the current suite required such a elaborate shell/scripting
environment and how that fit's in to the script which are executed.
Perhaps someone could write up a explanation for all the member of the
list -- so that we all understand who the piece are currently
constructed.  Any volunteers?


Finally, I would suggest that before we go writing any code that we need
to explore the options available to us.  The options which have been
mentioned so far include:

- modifying the current TCS suite
- PostgreSQL testing suite
- NIST suite
- XP test framework
- Open Source Database Benchmark (http://freshmeat.net/projects/osdb/)
- CppUnit 

Any other suggestions?


Sean

[Firebird-test] Firebird Testing - A plan for discussion

[Leyne, S. <sl...@at...>]

All,

After discussion amongst the Firebird Admin Team, I've taken on the role
as the Testing Project Co-coordinator (actually, Mark O'Donohue gave me
an incentive).

Of all aspects of the Firebird, testing has garnered the least attention
so far and is the least understood by most of the members.  With the
upcoming v1.0 release, this needs to change.

So, please find below my proposal (open for discussion) of the
plan/priorities for the long-term testing aspects of the Firebird
project.

Realistically, however, the main focus of the testing effort for
Firebird 1.0 will be on testing the database engine.  The structure of
the peripherals and even most of the engine still (despite bug fixes and
the occasional feature) does not differ remarkably from the original
code, so most of the testing done by Borland is still applicable to the
Firebird codebase.  As time goes on an Firebird evolves and begins to
differ from the InterBase original we expect the test suites to grow
with the project and eventually cover all the features of Firebird both
new and original.

Finally, an opensource and free Firebird project is only going to
succeed 
if people step up, participate and contribute.  In the current testing 
phase this is an opportunity to contribute for those of us who are to 
directly benefit from the Firebird project.  To that end I/we welcome
all volunteers.


Sean

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - -

Introduction

The current Firebird release policy requires that all releases (whether
final, test, dev, alpha or beta) successfully pass through the complete
TCS Suite as it exists today.

The TCS Suite, was released by Borland, and it covers a broad range of
the engine functionality and was developed of the course of many years.
So in principle all we need to do is compile the engine and run the
tests to check our changes.

But, as was the case with the engine source code, life is not quite that
simple.

The suite (like the source code) has suffered from some disrepair and
was designed to work within a very specific installation framework
within the Borland lab (especially the Win32 environment).  This has
meant that only the most knowledgeable can effectively build and run the
test suite.

A lot of the tests relate to aspects of the project that are no longer
heavily used, gdml for instance and are lacking in any functionality to
cover the newer ways that databases are used in more detail, the java
interface for instance.

Finally, Borland did not choose to release the stress and volume testing
routines which are used as part of "certification" process.

All, however, is not lost.

First, it is very important to remember that the Firebird project
members are proving every day that we have the technical expertise to
attack any problem (a c++ version of the engine anyone?) and that our
knowledge of the engine is growing.

There are a number of additional test suites we can draw on to
supplement the existing tests.  The PostgreSQL test suite, the NIST3 sql
coverage test suite (which was also used as a major? part of the
internal Borland QA suite), the TPS3 (??) benchmark testing suites and
the Sun jdbc compliance test suite.


1.	Creation of "install and go" Testing Binaries

In order for testing to become widely accepted and pursued, we need to
make the process of running the testing programs as simple as installing
the client/engine binaries themselves.

This extends to developing instructions on: the installation and
execution for the testing programs, reporting test failures to the
project team, building and submitting new scripts to the master project.


2.	Elimination of the database for storing scripts and results

(This is a personal pet peeve of mine -- I've never understood the logic
of need the engine to be running in order to test the engine!).

This will require a change to the testing program logic but I feel that
it is necessary for three reasons.

First, storing the scripts in the database means that changes to scripts
can't be easily tracked within the CVS structure.

Second, storing the scripts and results as simple TXT files (we can
argue about the format choice later) would eliminate the need for the
engine to be installed on the local PC (running the testing programs)
and thus would further reduce the footprint and 'bar' for those looking
to contribute to the testing process.

Finally, there are no tools to enable people to read/edit the scripts
within the database.  This makes the testing even more difficult for
community members to understand and therefore contribute to.


3.	Need to develop engine stress/load testing programs.

This is something which will require the most effort.

First, a clear statement of the goal needs to be developed.  The
information/requirements gathering and discussions are already underway.
All input is appreciated.

Second, an architecture/approach needs to be selected -- the approach
will probably change over time but we need to start somewhere (and
besides some of us have nothing but free time to re-architect the
programs <huge grin>).

It has been suggested, and agreed by a number of people, that if the new
stress/load testing routines are designed correctly, then it could also
form the basis of a test bed for a more general test suite, replacing
the current TCS program.

Finally, the programs need to be developed.


4.	Need to develop complete client data access testing programs.

There are no tests for any of the client data access tools, so any new
development which takes place on ODBC/ADO/JDBC, needs to go through a
testing regiment of it's own. 

This task should be the responsibility of each development team and be a
required part of their development plans -- so much so, that sucessfull
testing should be required prior to a release being made (ODBC/JBDC
projects are you listening???).


5.	Documentation of Testing tasks

This topic is a little wide ranging.

It starts with an update on the status of the testing programs for the
various platforms.  For example, can the tests be run against a remote
engine or only local engines, what is the status of the Win32 testing
install/environment (as far as I know it needs a lot of work). 

From there, we need to document and categorize all of the exiting
scripts, test groups and expected results.  

From there, we need to add new tests which attacks the bugs which have
been fixed in the engine as well as those bugs which have been found and
remain unresolved (this may require adding logic to handle expected test
failures).  [This could produce interesting results when run against the
Borland "certified" engine]

Re: [Firebird-test] FBT - firebird tests

[Pavel C. <pc...@at...>]

Hi,

On 12 May 2001, at 22:39, Tord Hammer wrote:

> I dont think, unit tests are possible with the current source code.
> Maybe its possible to add unit tests to the firebird 2 source.
> Pavel Cisar suggested something similar.

When I referred to the XP unit tests, I mean the use of XP test 
framework for black-box testing, i.e. as a foundation for TCS front-
end. I didn't even dreamed about the white-box testing at this stage 
(it would be great to have it in future, but this issue fall in scope to 
the fb-devel list).
 
> Since I think, only a replacement for the TCS make sense, the old
> tests should be included in the new test system.

Yes.
 
Best regards
-- Pavel

Re: [Firebird-test] FBT - firebird tests

[David J. <dav...@ea...>]

Hi,
I think TCS in the db is (conceptually) great except for two things:

1. No version control (obviously it could be implemented, but would take
work)

2. You have to download the whole thing to get a one line change (again,
can be fixed, but takes work).

Is this more or less work than a new system? I don't know.  It probably
couldn't be live on sourceforge without convincing them to run a fb server
for the purpose.  A good idea, but may not be easy.

david jencks
On 2001.05.12 13:31:34 -0400 Ann W. Harrison wrote:
> At 03:40 PM 5/12/2001 +0200, Tord Hammer wrote:
> 
> >I suggest a new test systems called FBT ;-) which works
> >like the PostgresQL one. It would be strict script-based,
> >which means all tests and expected results are stored
> >in text files. Groups of tests (calles series in TCS) and
> >group of series (calles meta-series) would be text files
> >too.
> 
> I guess I'm a bit confused by the resistance of a database
> development project to using database technology.
> 
> 
> Regards,
> 
> Ann
> www.ibphoenix.com
> We have answers.
> 
> 
> _______________________________________________
> Firebird-test mailing list
> Fir...@li...
> http://lists.sourceforge.net/lists/listinfo/firebird-test
> 

Re: [Firebird-test] FBT - firebird tests

[Tord H. <th...@ha...>]

Hi,

> Another possible influence to look at is xunit testing frameworks,
although
> I'm not sure our tests at this point are in any way unit tests.

The current TCS is a black-box-test system, which means, only the
output / behavior from the compiled firebird server is tested.
White-box-tests or unit tests dont exist at all.

<sarcasm>
"If architects would work like software engineers, a lonely
woodpecker could destroy the whole civilisation." (unknown)
</sarcasm>

I dont think, unit tests are possible with the current source code.
Maybe its possible to add unit tests to the firebird 2 source.
Pavel Cisar suggested something similar.

> One question I have about this approach is the extent to which we
wish to
> translate the TCS tests into the file based format, and how to do
this.

Since I think, only a replacement for the TCS make sense, the old
tests should be included in the new test system.

> I could probably find it pretty easily, but do you have a link to
the
> postgres system?

It's at www.postgresql.org. The tests are in the src/test directory
from the source tree.


Tord

Re: [Firebird-test] FBT - firebird tests

[David J. <dav...@ea...>]

Hi,

I agree separate files are essential.

I think some consideration should be given to the possibility of leveraging
xml processing to help us out.

For dynamic sql, I suggest that the statements to be executed be in a xml
file, one element /statement.  The results can also be in xml format.  It
is then pretty easy to include both the statement and the expected results
in one file, and use xml query (xsl) to find particular parts, browse, etc.

I am working on a java project along these lines, I should have something
to look at in a few days.  Right now it is mostly a scripting engine,
although it can compare new and reference results.  Slight modifications
would be necessary to use the xml format I propose above.

Another possible influence to look at is xunit testing frameworks, although
I'm not sure our tests at this point are in any way unit tests.

One question I have about this approach is the extent to which we wish to
translate the TCS tests into the file based format, and how to do this.

The tool I am working on can easily be used to generate an xml file
containing the contents of a TCS database.  I'm not sure what to do with it
next.

I could probably find it pretty easily, but do you have a link to the
postgres system?


Thanks!

david jencks

On 2001.05.12 09:40:57 -0400 Tord Hammer wrote:
> Hi,
> 
> up to now the only way to test the changes in firebird
> is the old (summer 2000) test suite from borland (aka TCS).
> 
> I suggest a new test systems called FBT ;-) which works
> like the PostgresQL one. It would be strict script-based,
> which means all tests and expected results are stored
> in text files. Groups of tests (calles series in TCS) and
> group of series (calles meta-series) would be text files
> too. 
> 
> In about two weeks I'll have enough spare time to start
> this project.
> 
> Ideas, thoughts ?
> 
> 
> Tord
> 
> 
> 
> _______________________________________________
> Firebird-test mailing list
> Fir...@li...
> http://lists.sourceforge.net/lists/listinfo/firebird-test
>