firebird-test

[Firebird-test] FBT - firebird tests

[Tord H. <th...@ha...>]

Hi,

up to now the only way to test the changes in firebird
is the old (summer 2000) test suite from borland (aka TCS).

I suggest a new test systems called FBT ;-) which works
like the PostgresQL one. It would be strict script-based,
which means all tests and expected results are stored
in text files. Groups of tests (calles series in TCS) and
group of series (calles meta-series) would be text files
too. 

In about two weeks I'll have enough spare time to start
this project.

Ideas, thoughts ?


Tord

RE: [Firebird-test] FBT - firebird tests

[Dmitry Y. <di...@ma...>]

Hi Tord,

> up to now the only way to test the changes in firebird
> is the old (summer 2000) test suite from borland (aka TCS).
>
> I suggest a new test systems called FBT ;-) which works
> like the PostgresQL one. It would be strict script-based,
> which means all tests and expected results are stored
> in text files. Groups of tests (calles series in TCS) and
> group of series (calles meta-series) would be text files
> too.
>
> In about two weeks I'll have enough spare time to start
> this project.
>
> Ideas, thoughts ?

Sounds good to me. I think it would be our another great step forward, so
feel free to start a stone rolling.

Cheers,
Dmitry

RE: [Firebird-test] FBT - firebird tests

[Pavel C. <pc...@at...>]

Hi all,

There was a talk about XP-like test suite some time ago. I think 
that it's the right way to go. For details, look at 
http://www.xprogramming.com/
There are numerous XP test frameworks for various languages. The 
only one thing that we have to solve is cross-platform compatibility 
issue. That mean that tests have to be developed in one from next 
languages: C/C++, Java, Python. Personally, I'd vote for Python.

Comments?

Regards
-- Pavel

Re: [Firebird-test] FBT - firebird tests

[Ann W. H. <aha...@ib...>]

At 03:40 PM 5/12/2001 +0200, Tord Hammer wrote:

>I suggest a new test systems called FBT ;-) which works
>like the PostgresQL one. It would be strict script-based,
>which means all tests and expected results are stored
>in text files. Groups of tests (calles series in TCS) and
>group of series (calles meta-series) would be text files
>too.

I guess I'm a bit confused by the resistance of a database
development project to using database technology.


Regards,

Ann
www.ibphoenix.com
We have answers.

Re: [Firebird-test] FBT - firebird tests

[<Sch...@t-...>]

"Ann W. Harrison" wrote:
> 
> At 03:40 PM 5/12/2001 +0200, Tord Hammer wrote:
> 
> >I suggest a new test systems called FBT ;-) which works
> >like the PostgresQL one. It would be strict script-based,
> >which means all tests and expected results are stored
> >in text files. Groups of tests (calles series in TCS) and
> >group of series (calles meta-series) would be text files
> >too.
> 
> I guess I'm a bit confused by the resistance of a database
> development project to using database technology.

Me too, maybe its just that these scripts from tcs are
too *n*x oriented. Or the front end is a bit outdated.
It's just plain text that is stored in the test db's.

IMHO we need 
1. an easy way to maintain the scripts under cvs control
2. a platform independent (modern) frontend to this stuff
3. a way to use this under MS Win_something 
   without Cygwin et al

And time to do this all :-(

This doesn't mean that I wouldn't be happy about any 
other solution that solves the problem. (especially
for the MS platforms)

So, never mind.


Frank
-- 
"Fascinating creatures, phoenixes. They can carry immensely heavy loads,
 their tears have healing powers and they make highly faithful pets."
         - J.K. Rowling
http://firebird.sourceforge.net

Re: [Firebird-test] FBT - firebird tests

[Tord H. <th...@ha...>]

Hi,

> I guess I'm a bit confused by the resistance of a database
> development project to using database technology.

The TCS needs a central database server which holds the global
test database. The local database file is (according to the
docs) only needed if the QA person needs to create a new test
and want do try something without screwing the main global
database.

Second I assume the majority of test runs in the borland QA
are tested against a test database server (remote). I dont
think the QA members installed an unstable version on there
workstations, when the TCS needs a database server to access
the test files.

Third maintaining the global test database is now impossible
with the current firebird structure. And no, I dont think
fetch from cvs, restore gdb, change test, backup gdb,
store to cvs is a reasonable way to create and maintain
tests.

Most people who want to help testing dont have multiple
computers or a client-server-setup like the QA office at
borland.

I have monitored the firebird progress since the first days
and in seven months only one (the generator test from FSG) 
test where added to the original TCS.

So there are NO scripts to test the new features of firebird.
NOBODY can test if the latest changes (e.g. metadata issues
from Pattrick Griffin, missing services entry fallback, 64 bit
io, 16k pages) workes, they can only test that the changes
dont break the old behavior.

And there are NO scripts to test the error fixes. The firebird
team (mainly Ann and Skywalker) have fixed a bunch of typing
error (like a == b insteed of a = b, or missing parentheses),
but there are no test scripts which failed before the fix
and pass now. So you just dont know if you fix it correctly
or screw up a side-effect of this typing error.

Last but not least, there are no test scripts which shows the
bug reports at Sourceforge. A metaseries SF_BUGS which
fails at current build but passes with firebird 1.0 would
be a very satisfied moment, wouldn't it.

For me this shows, that either nobody is willing to create
tests or nobody is able to create tests. I hope it's the
second one ;-)

Just thoughts,
Tord

Remark:
English is not my native language (hint: look at TLD), so
if some words sound insulting, it's only because these
words were the first ones in my dictionary <g>

Re: [Firebird-test] FBT - firebird tests

[David J. <dav...@ea...>]

Hi,
I think TCS in the db is (conceptually) great except for two things:

1. No version control (obviously it could be implemented, but would take
work)

2. You have to download the whole thing to get a one line change (again,
can be fixed, but takes work).

Is this more or less work than a new system? I don't know.  It probably
couldn't be live on sourceforge without convincing them to run a fb server
for the purpose.  A good idea, but may not be easy.

david jencks
On 2001.05.12 13:31:34 -0400 Ann W. Harrison wrote:
> At 03:40 PM 5/12/2001 +0200, Tord Hammer wrote:
> 
> >I suggest a new test systems called FBT ;-) which works
> >like the PostgresQL one. It would be strict script-based,
> >which means all tests and expected results are stored
> >in text files. Groups of tests (calles series in TCS) and
> >group of series (calles meta-series) would be text files
> >too.
> 
> I guess I'm a bit confused by the resistance of a database
> development project to using database technology.
> 
> 
> Regards,
> 
> Ann
> www.ibphoenix.com
> We have answers.
> 
> 
> _______________________________________________
> Firebird-test mailing list
> Fir...@li...
> http://lists.sourceforge.net/lists/listinfo/firebird-test
> 

Re: [Firebird-test] FBT - firebird tests

[David J. <dav...@ea...>]

Hi,

I agree separate files are essential.

I think some consideration should be given to the possibility of leveraging
xml processing to help us out.

For dynamic sql, I suggest that the statements to be executed be in a xml
file, one element /statement.  The results can also be in xml format.  It
is then pretty easy to include both the statement and the expected results
in one file, and use xml query (xsl) to find particular parts, browse, etc.

I am working on a java project along these lines, I should have something
to look at in a few days.  Right now it is mostly a scripting engine,
although it can compare new and reference results.  Slight modifications
would be necessary to use the xml format I propose above.

Another possible influence to look at is xunit testing frameworks, although
I'm not sure our tests at this point are in any way unit tests.

One question I have about this approach is the extent to which we wish to
translate the TCS tests into the file based format, and how to do this.

The tool I am working on can easily be used to generate an xml file
containing the contents of a TCS database.  I'm not sure what to do with it
next.

I could probably find it pretty easily, but do you have a link to the
postgres system?


Thanks!

david jencks

On 2001.05.12 09:40:57 -0400 Tord Hammer wrote:
> Hi,
> 
> up to now the only way to test the changes in firebird
> is the old (summer 2000) test suite from borland (aka TCS).
> 
> I suggest a new test systems called FBT ;-) which works
> like the PostgresQL one. It would be strict script-based,
> which means all tests and expected results are stored
> in text files. Groups of tests (calles series in TCS) and
> group of series (calles meta-series) would be text files
> too. 
> 
> In about two weeks I'll have enough spare time to start
> this project.
> 
> Ideas, thoughts ?
> 
> 
> Tord
> 
> 
> 
> _______________________________________________
> Firebird-test mailing list
> Fir...@li...
> http://lists.sourceforge.net/lists/listinfo/firebird-test
> 

Re: [Firebird-test] FBT - firebird tests

[Tord H. <th...@ha...>]

Hi,

> Another possible influence to look at is xunit testing frameworks,
although
> I'm not sure our tests at this point are in any way unit tests.

The current TCS is a black-box-test system, which means, only the
output / behavior from the compiled firebird server is tested.
White-box-tests or unit tests dont exist at all.

<sarcasm>
"If architects would work like software engineers, a lonely
woodpecker could destroy the whole civilisation." (unknown)
</sarcasm>

I dont think, unit tests are possible with the current source code.
Maybe its possible to add unit tests to the firebird 2 source.
Pavel Cisar suggested something similar.

> One question I have about this approach is the extent to which we
wish to
> translate the TCS tests into the file based format, and how to do
this.

Since I think, only a replacement for the TCS make sense, the old
tests should be included in the new test system.

> I could probably find it pretty easily, but do you have a link to
the
> postgres system?

It's at www.postgresql.org. The tests are in the src/test directory
from the source tree.


Tord

Re: [Firebird-test] FBT - firebird tests

[Luis F. <lou...@no...>]

Hello,

In developing the test scripts does it matter what environment is used.
I was thinking perhaps of a Delphi/Kylix app to serve as  a framework. 
Of course, that leaves out Solaris and others. Whatever the front end
app is,  I think that by keeping the tests in a format that is easy to
get to and use, several different tools could be developed by users for
either Open Source or internal testing.  I have two extra machines that
I plan on loading Linux on for testing interbase and have not had the
time to figure TCS out. I would be glad to get involved in the testing
effort.

- Lou

Tord Hammer wrote:
> 
> Hi,
> 
> up to now the only way to test the changes in firebird
> is the old (summer 2000) test suite from borland (aka TCS).
> 
> I suggest a new test systems called FBT ;-) which works
> like the PostgresQL one. It would be strict script-based,
> which means all tests and expected results are stored
> in text files. Groups of tests (calles series in TCS) and
> group of series (calles meta-series) would be text files
> too.
>