firebird-test — Firebird engine testing, test flight/suite discussion/information

Re: [Firebird-test] Testing Firbird --- SI-T1.0.0.366 Firebird Be

[Mark O'D. <mar...@lu...>]

Hi Sean

Leyne, Sean wrote:

>
>What would it take to have the reports produce a more 'intelligible'
>output?
>
>This output seems absolutely useless!
>
It is, what John is working on will enable us to do a unified diff (aka 
the code changes), and to mix the stdout/stderror streams so the output 
appears normal and in context.  

It's not far off, last report was there were only 8 TCS tests that when 
converted still produced different results to the TCS run - but I should 
let him describe that.

Cheers

Mark

-- 
Your database needs YOU!
http://www.firebirdSQL.org

RE: [Firebird-test] Testing Firbird --- SI-T1.0.0.366 Firebird Be

[Leyne, S. <sl...@at...>]

All,

> Welcome to the joys of decipering TCS, it looks pretty much like mine 
> did, when I extracted the scripts and
> ran them and looked at the output and expected output, the 
> difference is 
> the new stuff is sorted on City, then something then something else. 
>  Hence in the new one the results are in order Atlanta, 
> Boston, Chicago...
> The same test is run a few times (isql embedded etc), when 
> you look at 
> the raw output, you can see it's just the sort order that is 
> different.

<snip output from isc_diff>

What would it take to have the reports produce a more 'intelligible'
output?

This output seems absolutely useless!


Sean

[Firebird-test] TCS Install for Win32

[Leyne, S. <sl...@at...>]

I am wondering what it would take to build a TCS Install for Win32, to
allow for the TCS suite to be downloaded and run with a 'minimum'
footprint?

We would get me people involved in testing if the TCS process was more
of an "install and go" task.

I know that a number of the tests scripts involve using GPRE and a
compiler to create an EXE to generate the results.  Could this be
'replaced' with pre-built EXE?  (Thus eliminating the need for a C
compiler)

I also know that TCS will require some Cygwin elements -- Can someone
but together a list of them to minimize the required footprint.

(Ooops, this should have been the first question)  Can TCS run on a
local database?


Sean

Re: [Firebird-test] Testing Firbird --- SI-T1.0.0.366 Firebird Be

[Mark O'D. <mar...@lu...>]

Konstantin Kuznetsov wrote:

>Hi all!
>
>>Could you post the errors/results for our review...  a number of the
>>errors are due to Claudio V's improved error reporting, so they are not
>>real errors.
>>
>I thing your mean display data that appears on screen.
>Next some screan shots. Dont know how get file report.
>
>
> r c_sql_join_4
>Running global test C_SQL_JOIN_4 ... *** failed ****
>Test C_SQL_JOIN_4 failed on 22-AUG-2001:
>        initialized by FSG
>        on 16-OCT-2000 with boilerplate QA_BP
>

Welcome to the joys of decipering TCS, it looks pretty much like mine 
did, when I extracted the scripts and
ran them and looked at the output and expected output, the difference is 
the new stuff is sorted on City, then something then something else. 
 Hence in the new one the results are in order Atlanta, Boston, Chicago...
The same test is run a few times (isql embedded etc), when you look at 
the raw output, you can see it's just the sort order that is different.

>
>-------------------
>3 a 3,8
>
>>Braves               Atlanta               Georgia
>>Red Sox              Boston                Massachusetts
>>Cubs                 Chicago               Illinois
>>White Sox            Chicago               Illinois
>>
>skip table
>4 c 10,11
>tbl
>6 a 13,19
>tbl
>7,20 d 21
>tbl
>++++++++++++++++++++++++++++++++++++++++++++++++++
>27 a 27,32
>28 c 34,35
>< Braves          Atlanta                   Georgia
>---------------------------------------------
>
>>Royals          Kansas City               Missouri
>>Dodgers         Los Angeles               California
>>
>++++++++++++++++++++++++++++++++++++++++++++++++++
>30 a 37,43
>...
>
>>Mariners        Seattle                   Washington
>>
>++++++++++++++++++++++++++++++++++++++++++++++++++
>31,45 c 45
>
>++++++++++++++++++++++++++++++++++++++++++++++++++
>47,48 c 47,48
><
>------
>52 a 52,57
>
>53 c 59,60
>< Braves          Atlanta                   Georgia
>---------------------------------------------
>
>>Royals          Kansas City               Missouri
>>Dodgers         Los Angeles               California
>>
>++++++++++++++++++++++++++++++++++++++++++++++++++
>55 a 62,68
>++++++++++++++++++++++++++++++++++++++++++++++++++
>56,70 d 70
>< Cubs
>
>71 a 70
>________________________
>tcs> r dsql_domain_08
>Running global test DSQL_DOMAIN_08 ... *** failed ****
>Test DSQL_DOMAIN_08 failed on 22-AUG-2001:
>        initialized by DSMIRNOV
>        on 4-NOV-1999 with boilerplate RD_VECTOR
>
>
>-------------------
>1 c 1
>---------------------------------------------
>++++++++++++++++++++++++++++++++++++++++++++++++++
>7 c 7
>< -Token unknown - line 4, char 15
>---------------------------------------------
>
>>-Token unknown - line 3, char 17
>>
>++++++++++++++++++++++++++++++++++++++++++++++++++
>

This rings a bell too, and the error, but I can't remember what I found 
when I looked at it in detail... ok I looked again, if you look at the 
script and error one stmt should fail, it does, so the test works correctly.

I assume the difference in error position has resulted from some work 
Claudio did on improving the error hadling to corretly identify the line 
and column, apparently it used to be wrong in some cases.


The sequence is:

create domain dom08a_1 as smallint;
commit;
show domain dom08a_1;

/* the next statement should fail */

alter domain dom08a_1
 add constraint not null;
commit;


The expected output is:


SQL> SQL> SQL> DOM08A_1                        SMALLINT Nullable
SQL> SQL> CON> CON> CON> Statement failed, SQLCODE = -104

Dynamic SQL Error
-SQL error code = -104
-Token unknown - line 4, char 15
-not


The only difference is it now reports the error at:

-Token unknown - line 3, char 17


The 'not' is in column 17, buggered if I can figure out the correct line 
number :-), tcs is a bit messy, but 15's not correct.  So it looks 
pretty reasonable.  But having produced an error, the correct results of 
the test has been obtained.

[snip]

>Who can know of his place?
>From c_dsql_ri_alt_add
>tcs.script
>
>rm -f sh_test.h
>gbak -r ./tests/sov3v4files/atlas.gbak
>/export/home/interbase/TSC.10/tcs/scripts
>ri_views /export/home/interbase/TSC.10/tcs/scripts/atlas.gdb
>cp  /TCS/sh_test.h sh_test.h
>cat >  al
>
>There I can find programm or script "ri_views "?
>I have not such thing!
>
I had that as well, ri_views is created by either the test directly 
before it or  the first one in the series.  It is deleted by the last 
one in the series I think.

I extracted the data using Franks extract kit, and then ran these using 
the trial dejagnu test kit, which made it a little easier to get at the 
intermediate files.

So all in all, your looking pretty good, there :-).

Cheers

Mark



-- 
Your database needs YOU!
http://firebird.sourceforge.net
GPL: free software today - and still free tomorrow

Re: [Firebird-test] Testing Firbird --- SI-T1.0.0.366 Firebird Be

[Konstantin K. <klk...@ns...>]

Hi all!
> Could you post the errors/results for our review...  a number of the
> errors are due to Claudio V's improved error reporting, so they are not
> real errors.
I thing your mean display data that appears on screen.
Next some screan shots. Dont know how get file report.


 r c_sql_join_4
Running global test C_SQL_JOIN_4 ... *** failed ****
Test C_SQL_JOIN_4 failed on 22-AUG-2001:
        initialized by FSG
        on 16-OCT-2000 with boilerplate QA_BP
-------------------
3 a 3,8
> Braves               Atlanta               Georgia
> Red Sox              Boston                Massachusetts
> Cubs                 Chicago               Illinois
> White Sox            Chicago               Illinois
skip table
4 c 10,11
tbl
6 a 13,19
tbl
7,20 d 21
tbl
++++++++++++++++++++++++++++++++++++++++++++++++++
27 a 27,32
28 c 34,35
< Braves          Atlanta                   Georgia
---------------------------------------------
> Royals          Kansas City               Missouri
> Dodgers         Los Angeles               California
++++++++++++++++++++++++++++++++++++++++++++++++++
30 a 37,43
...
> Mariners        Seattle                   Washington
++++++++++++++++++++++++++++++++++++++++++++++++++
31,45 c 45
>
++++++++++++++++++++++++++++++++++++++++++++++++++
47,48 c 47,48
<
------
52 a 52,57

53 c 59,60
< Braves          Atlanta                   Georgia
---------------------------------------------
> Royals          Kansas City               Missouri
> Dodgers         Los Angeles               California
++++++++++++++++++++++++++++++++++++++++++++++++++
55 a 62,68
++++++++++++++++++++++++++++++++++++++++++++++++++
56,70 d 70
< Cubs

71 a 70
________________________
tcs> r dsql_domain_08
Running global test DSQL_DOMAIN_08 ... *** failed ****
Test DSQL_DOMAIN_08 failed on 22-AUG-2001:
        initialized by DSMIRNOV
        on 4-NOV-1999 with boilerplate RD_VECTOR
-------------------
1 c 1
---------------------------------------------
++++++++++++++++++++++++++++++++++++++++++++++++++
7 c 7
< -Token unknown - line 4, char 15
---------------------------------------------
> -Token unknown - line 3, char 17
++++++++++++++++++++++++++++++++++++++++++++++++++

FB_SQL - series are PASSED . My error was no file
./test/sov3files/v5/ib_udf.sql

Who can know of his place?
From c_dsql_ri_alt_add
tcs.script

rm -f sh_test.h
gbak -r ./tests/sov3v4files/atlas.gbak
/export/home/interbase/TSC.10/tcs/scripts
ri_views /export/home/interbase/TSC.10/tcs/scripts/atlas.gdb
cp  /TCS/sh_test.h sh_test.h
cat >  al

There I can find programm or script "ri_views "?
I have not such thing!


Konstantin

RE: [Firebird-test] Testing Firbird --- SI-T1.0.0.366 Firebird Be ta2

[Leyne, S. <sl...@at...>]

Konstantin,

Could you post the errors/results for our review...  a number of the
errors are due to Claudio V's improved error reporting, so they are not
real errors.

I think the "C_DSQL_RI_ALT_ADD" is due to a change that Ann made to the
indexing structure for the system tables -- this will likely get backed
out.

I can't comment about the event errors.


Sean

[Firebird-test] Testing Firbird --- SI-T1.0.0.366 Firebird Beta2

[Konstantin K. <klk...@ns...>]

Hi All !
I just build  Intel Solaris port with updated tsc version 4.16
(yesterday download) and follows tests failed:
c_sql_join_4
dsql_domain_08
fb_sql
c_dsql_ri_alt_add

And follows as usial wait() forever linked with "-lthread"
event_misc_1
sql_event_13

Any suggestions ?

--
Konstantin Kuznetsov, phD

Re: [Firebird-test] Small install mistake...

[David R. R. <ibi...@ma...>]

> just download FB Win32 1-beta2 and noticed the installer telling me
it's
> installing the 'interbase developer tools' and 'interbase examples'
while
> copying files :)

Good catch.  I missed the file copy descriptions.  I think I'll just
change them to be "Examples" and "Developer Tools" so I don't have to
distinguish Firebird and InterBase.  I use the same script for the
InterBase and Firebird installs since 95% of what the scripts do is
exactly the same for both.

--
David R. Robinson
InterBase Installation Info - http://ibinstall.defined.net

[Firebird-test] Re: [Firebird-devel] TPC-R load / run test

[Mark O'D. <mar...@lu...>]

I write to you under difficult circumstance, my machine is slugishly
slow, there is next to no space left in my /tmp directory, and the
characters are slow to appear on the screen as I type.  What does this mean?

Well it means I've been able to run Tord's TPC-R benchmarks (mind you I
should have read the instructions I didn't have 2gig spare on my root
partition).  fortunately it came with a scale option so I could get by
with somewhat less.

I get a few errors, but basically it's there, next week (moving on from
dejaTCS testing, and TCS testing of 1.0.0. Beta2 linux builds) I will be
back to looking at the testing the >2Gbyte databases for fb linux
edition.  I have a (spare?) dual 1ghz machine there with a 40gig disk
space that I can use to test them, and Im sure these test are something
I can use...

Thanks Tord for giving me the ability to trash my linux box :-).

Cheers

Mark



Tord Hammer wrote:

 >Hi,
 >
 >I have tried to get the TPC-R benchmark running with FireBird.
 >
 >You can download the linux-version from:
 >http://www.hammer-software.de/fb/tpcr-firebird.tar.gz
 >


 >
 >The original sourcecode can be found at
 >http://www.tpc.org/tpcr/default.asp#spec
 >
 >If you want only the original queries than look at
 >http://www.hammer-software.de/fb/tpcr-queries.tar.gz
 >
 >
 >After extracting of tpcr-firebird.tar.gz, you can generate
 >the tpcr.gdb with the command:
 >    ./maketpc.sh
 >
 >Useful parameters are -h for help, -p for path-prefix (if you
 >dont have enough room at /tmp) and -s for scale factor.
 >
 >The queries can be executed with
 >    ./runtpc.sh
 >
 >Now my results:
 >
 >with scale 0.1 (results in a 150 MB database), the queries need
 >about 5-10 seconds each, with 100-200 seconds for query 20 and 21.
 >
 >for scale 1.0 and the parameter-values from tpcr_current.pdf
 >I could check the returned rows as well:
 >
 >maketpc run about two hours.
 >
 >query  | execution time (seconds) | correct result
 >-------+--------------------------+---------------
 >   1   |          700             |       yes
 >   2   |          250             |       yes
 >   3   |          700             |        no
 >   4   |          300             |       yes
 >   5   |          300             |       yes
 >   6   |          300             |       yes
 >   7   |          700             |       yes
 >   8   |          500             |       yes
 >   9   |         6500             |       yes
 >  10   |          400             |        no
 >-------+--------------------------+---------------
 >  11   |          150             |       yes
 >  12   |          350             |        no
 >  13   |         1800             |        no
 >  14   |          200             |       yes
 >  15   |          570             |       yes
 >  16   |           70             |       yes
 >  17   |          150             |        no
 >  18   |          350             |       yes
 >  19   |          700             |       yes
 >  20   |       still running      |       n/a
 >-------+--------------------------+---------------
 >  21   |       still running      |       n/a
 >  22   |       still running      |       n/a
 >
 >
 >The system is linux 2.2.16 (from SuSE 7.0)
 >AthlonB 1200, 256 GB RAM
 >one big 20 GB ext2-partition at the second half of a
 >40 GB hard disk.
 >FireBird Classic Build 347
 >Buffers = 10000
 >
 >
 >Comments, better queries or better indexes welcome.
 >
 >
 >Tord
 >
 >
 >
 >
 >_______________________________________________
 >Firebird-devel mailing list
 >Fir...@li...
 >http://lists.sourceforge.net/lists/listinfo/firebird-devel
 >


-- 
Your database needs YOU!
http://firebird.sourceforge.net
GPL: free software today - and still free tomorrow

[Firebird-test] Small install mistake...

[Martijn T. <m.t...@up...>]

Hi,

just download FB Win32 1-beta2 and noticed the installer telling me it's
installing the 'interbase developer tools' and 'interbase examples' while
copying files :)

--
Martijn Tonies
InterBase Workbench - the developer tool for InterBase and Firebird
http://www.interbaseworkbench.com

Upscene Productions
http://www.upscene.com

"This is an object-oriented system.
If we change anything, the users object."

Re: [Firebird-test] What to do with TCS

[Frank Schlottmann-G. <sch...@t-...>]

Mark O'Donohue wrote:

> John Bellardo wrote:
> 
> >
> >
> > But we need to decide what to do with TCS first...
> 
> Well Im all for giving it the flick, and extracting the tests into CVS
> as flat files :-).
> 
> But I'd really like to hear what Paul Reeves has to say (and probably
> Frank), since they are the ones that have really done some worked with
> it.  And I suspect they will have found some merrit in keeping it.

Sorry for being a bit late, I had no time the last weeks ( and won't
have too much for some more weeks as it seems)
TCS was there when I started, so I tried to make it usable. I'm happy with 
any other solution (especially if it works better with CVS). 


Cheers 

Frank

-- 
"Fascinating creatures, phoenixes. They can carry immensely heavy loads,
 their tears have healing powers and they make highly faithful pets."
         - J.K. Rowling
http://firebirdsql.org

[Firebird-test] Tagged

[Mark O'D. <mar...@lu...>]

Hi

Im a fair way though validating the linux builds through the tcs suite. 
  And am fairly happy that we haven't dented the product too much.  I 
now get one error from the test suite,  C_DSQL_RI_ALT_ADD and even that 
seems to be related to a fix.

There is a bit more work to be done, but Im feeling comfortable enough 
that there are no major problems so I've marked the version in the CVS 
tree as T1_0_0_Beta2 from 8-Aug-2001.

The linux builds should come out shortly.

If any of the build platforms need to sneek something in, or need a 
change to be applied to the build, we can do it a few ways, (probably 
commit it as normal, then we will release src + patch through 
sourceforge with the binary) anyway contact me to see what we do.

There is a bit of activity going on in testing at the moment, from a 
number of different sources, (good to see :-), and that will result in a 
RC1 Release Candiate build.  Im reluctant to say exacty how long, since 
it's all on peoples spare time, and I can't speak on their behalf.  But 
I believe that within 3mths version 1.0 will be out the door.

As a matter of tidying up, I've also put a tag in retrospectively for 
T0_9_5_Beta1, (18-May-2001 was the date)  but the build number will 
define exactly what version you have.

Cheers

Mark

-- 
Your database needs YOU!
http://firebird.sourceforge.net
GPL: free software today - and still free tomorrow

Re: [Firebird-test] Re: [Firebird-devel] TPC-R load / run test

[Mark O'D. <mar...@lu...>]

Tord Hammer wrote:

>
>The database server was a server with 16 nodes, each containing
>4 pentium iii/550 and 4 gb ram. The TOC for 5 years (including system
>price and support for 5 years) was about 13 million dollars :)
>
Shwing off :-), I seem to remember an article that NEC TerraData had 
actually yet to sell any of their high end multiprocessor nt boxes, 
perhaps the price tag explains that.

Thanks Tord, I certainly will enjoy having a look at this in the next 
few days.

Cheers

Mark

-- 
Your database needs YOU!
http://firebird.sourceforge.net
GPL: free software today - and still free tomorrow

Re: [Firebird-test] New deja-testsuite entry to play around with :-).

[John B. <bel...@cs...>]

Hi,

On Monday, August 13, 2001, at 10:11 PM, Mark O'Donohue wrote:

>
> Hi John
>
> I've commited the current test setup for the dejagnu test suite in 
> TCS/deja-testsuite,  mostly it;s the dejagnu tcs.exp file that John 
> wrote that parses the tcs database from flat files runs the tests adn 
> compares the output.
>
> If nothing else it should what you can do with a good scriping 
> language, (simulating tcs being a reasonably complex thing to do :-).

I was hoping it would provide a degree of backwards compatibility with 
new TCS tests released by Borland.  In the long run we should consider 
moving the tests outside the TCS framework for performance, 
maintainability, and ease of locating the bug.

> Mind you in Johns hands, I think if he put his mind to it, he's capable 
> of bending dos scripts round to doing it :-).
>

Uh oh, you want me to go and learn DOS scripting too?  :-)

>
> What I've done is gone for a two step process, extract current files to 
> flat files, (one step beyond the extract utlity) and then the run kit.
>
> Mostly it works on linux, and a fair chunk of it works on win32, I just 
> need to get the compiler flags etc set correctly to be able to run the 
> win32 compiled tests.

My original work was functioning on Darwin and I suspect Mark's 
modifications didn't break it much.

>
> It's all still experimental, and  but it does look pretty promising.

-John

Re: [Firebird-test] Re: [Firebird-devel] TPC-R load / run test

[Tord H. <th...@ha...>]

> Personally Im more interested in the testing perspective, in that it
> allows us to check for problems, but I assume the figures also allow
us
> to compare to other systems.  Any idea how we fare?

This is hard to guess, since the small "databases" (like postgres,
mysql,
and so on) didnt run the tpc benchmarks. And the results at tpc.org
use database sizes which are nearly impossible to handle with
firebird.
So I think, it can only be used to stress test the engine, when I
found
the reason, why the queries 3, 10, 12, 13 and 17 returned the wrong
result set.

But back to the tpc-r:

There are only two results submited at the tpc.org homepage. I have
downloaded the last result, dated March 6, 2000.

The benchmark run at scale 1000, which means 1000 Gigabytes worth of
data.

With only one query executing (== power test), all queries finished
after 3 hours 27 minutes.

In the second test (== throuhput test) 7 parallel thread were
executed.
Everyone did the full set of 22 queries. At the same time, a 8rd
thread
did 1,500,000 inserts into the orders table and about 5,250,000
inserts
into the lineitem table, after that the thread delete 1,500,000
entries
in the orders table together with the linked entries in the lineitem
table.

The 7 threads finished after about 6-9 hours, the 8rd thread
after 21 hours.

Load time for the database (including sending 1000 Gigabytes to the
database server and building all the indexes) was 84 hours 15 minutes.

The database server was a server with 16 nodes, each containing
4 pentium iii/550 and 4 gb ram. The TOC for 5 years (including system
price and support for 5 years) was about 13 million dollars :)



There are more results for the TPC-C benchmark, but this one is quite
difficult to convert, since this benchmark simulates terminals (with
think and key time). But FYI, I have downloaded the result from
the best non-cluster system as well.

In this test, a compaq Alphaserver with 32 cpus and 256 GB memory and
22 Terrabytes disk-space could handle 27 millions new-order (keying
in a new order from a customer) and 26 millions payment (commit
paying of a bill) transactions in two hours.

Remember the bug with transaction numbers, Ann fixed some days ago ?
With 1k pages the limit was about 130 million transactions; this
beast would hit the limit in about 4 hours :))


A little update to my last message:

> >query  | execution time (seconds) | correct result
> >-------+--------------------------+---------------
> >  20   |       still running      |       n/a

after 4 hours, this query runs still at 100% cpu and
nearly no disk activity.


Tord

[Firebird-test] Re: [Firebird-devel] TPC-R load / run test

[Mark O'D. <mar...@lu...>]

Hi Tord

That sounds bloody terrific, a thrashing program one of the key testing 
bits we were missing.  And it helps to have it based on the standards stuff.

I've cc'ed this to firebird-test and will definately have a good look at 
it in the next couple of days.

Personally Im more interested in the testing perspective, in that it 
allows us to check for problems, but I assume the figures also allow us 
to compare to other systems.  Any idea how we fare?

Cheers

Mark




Tord Hammer wrote:

>Hi,
>
>I have tried to get the TPC-R benchmark running with FireBird.
>
>
>The homepage is at http://www.tpc.org/tpcr/
>
>I have changed the dbgen utility to work with interbase and fixed
>some bugs in the progress indicator. 
>
>I have also changed all the queries.
>
>I have removed the parameters and inserted the values given in
>tpcr_current.pdf.
>I have changed the inner-select syntax to using a view.
>I have changed the case syntax to a stored procedure.
>I have also changed some queries which confused the optimizer.
>I have tried to find suitable indexes.
>
>Since I dont have included the important parts of the benchmark
>(refreshing test, throughput test, power test) and I have changed
>the queries too much, so you cant use it to get a correct benchmark
>result.
>
>But I think its good enough to stress test the engine.
>
>You can download the linux-version from:
>http://www.hammer-software.de/fb/tpcr-firebird.tar.gz
>
>The original sourcecode can be found at 
>http://www.tpc.org/tpcr/default.asp#spec
>
>If you want only the original queries than look at
>http://www.hammer-software.de/fb/tpcr-queries.tar.gz
>
>
>After extracting of tpcr-firebird.tar.gz, you can generate
>the tpcr.gdb with the command:
>    ./maketpc.sh
>
>Useful parameters are -h for help, -p for path-prefix (if you
>dont have enough room at /tmp) and -s for scale factor.
>
>The queries can be executed with
>    ./runtpc.sh
>
>Now my results:
>
>with scale 0.1 (results in a 150 MB database), the queries need
>about 5-10 seconds each, with 100-200 seconds for query 20 and 21.
>
>for scale 1.0 and the parameter-values from tpcr_current.pdf
>I could check the returned rows as well:
>
>maketpc run about two hours.
>
>query  | execution time (seconds) | correct result
>-------+--------------------------+---------------
>   1   |          700             |       yes
>   2   |          250             |       yes
>   3   |          700             |        no
>   4   |          300             |       yes
>   5   |          300             |       yes
>   6   |          300             |       yes
>   7   |          700             |       yes
>   8   |          500             |       yes
>   9   |         6500             |       yes
>  10   |          400             |        no
>-------+--------------------------+---------------
>  11   |          150             |       yes
>  12   |          350             |        no
>  13   |         1800             |        no
>  14   |          200             |       yes
>  15   |          570             |       yes
>  16   |           70             |       yes
>  17   |          150             |        no
>  18   |          350             |       yes
>  19   |          700             |       yes
>  20   |       still running      |       n/a
>-------+--------------------------+---------------
>  21   |       still running      |       n/a
>  22   |       still running      |       n/a
>
>
>The system is linux 2.2.16 (from SuSE 7.0)
>AthlonB 1200, 256 GB RAM
>one big 20 GB ext2-partition at the second half of a
>40 GB hard disk.
>FireBird Classic Build 347
>Buffers = 10000
>
>
>Comments, better queries or better indexes welcome.
>
>
>Tord
>
>
>
>
>_______________________________________________
>Firebird-devel mailing list
>Fir...@li...
>http://lists.sourceforge.net/lists/listinfo/firebird-devel
>


-- 
Your database needs YOU!
http://firebird.sourceforge.net
GPL: free software today - and still free tomorrow

[Firebird-test] New deja-testsuite entry to play around with :-).

[Mark O'D. <mar...@lu...>]

Hi John

I've commited the current test setup for the dejagnu test suite in 
TCS/deja-testsuite,  mostly it;s the dejagnu tcs.exp file that John 
wrote that parses the tcs database from flat files runs the tests adn 
compares the output.

If nothing else it should what you can do with a good scriping language, 
(simulating tcs being a reasonably complex thing to do :-).  

Mind you in Johns hands, I think if he put his mind to it, he's capable 
of bending dos scripts round to doing it :-).


What I've done is gone for a two step process, extract current files to 
flat files, (one step beyond the extract utlity) and then the run kit.

Mostly it works on linux, and a fair chunk of it works on win32, I just 
need to get the compiler flags etc set correctly to be able to run the 
win32 compiled tests.

It's all still experimental, and  but it does look pretty promising.


Cheers

Mark

Re: [Firebird-test] What to do with TCS

[John B. <bel...@cs...>]

Sean,

On Thursday, August 9, 2001, at 08:29 AM, Leyne, Sean wrote:

> John,
>
>>
>> But we need to decide what to do with TCS first...
>>
>
> Personally, right at the moment given the timelines that we need to
> follow -- my vote is to modify TCS to use flat/XML files (so at least we
> get rid of the DB dependency) and _perhaps_ update the code so that it
> can compile under c++ with autoconf (just like the v2 engine work).

Would this modification entail full support for update the TCS files?  
Read-only (from TCS's perspective) flat files are much easier to do than 
read-write.  If full write support was needed we could do it after v1 
was shipped.

>
> Once we have v1 out the door, we can start a broader review of the
> various approaches.

I see a slight difficulty here.  Once we have certified v1 efforts will 
rightly shift to the v2 codebase.  At that point developers are going to 
be much less willing to spend time working on the test suite because it 
is not a priority.  Having an integrated test suite will help with v2 
code base development, especially when we start making more critical 
changes.  Right now testing is a priority and there are a number of 
hands ready/willing to work on it.

-John

Re: [Firebird-test] What to do with TCS

[John B. <bel...@cs...>]

On Thursday, August 9, 2001, at 08:14 AM, Mark O'Donohue wrote:

> John Bellardo wrote:
>>
>> But we need to decide what to do with TCS first...
>
> Well Im all for giving it the flick, and extracting the tests into CVS 
> as flat files :-).
>
> But I'd really like to hear what Paul Reeves has to say (and probably 
> Frank), since they are the ones that have really done some worked with 
> it.  And I suspect they will have found some merrit in keeping it.

It does seem like a decent tool for handling the cross platform issues.  
The isql/qli scripts are not a big deal, but it does correctly handle 
c/c++ compilation which is a bonus.

>
> One think I have been thinking off, is a modified tcs program that 
> works from flat files could still be run from dejagnu, with perhaps an 
> option to use dejagnu to filter and compare the output.

Right, I was toying with that idea as well.  We could have a 
non-interactive mode for TCS where it would just do all the macro 
substitutions (and the like) it does now.  Then take the output script 
and run it with expect/tcl.  It would make a number of things that 
aren't working 100% in tcl work fine.  And we still have a TCS program 
to distribute to users (without the need for dejagnu) that they can use 
to perform some tests.

>
> TCS seems to me to represent a fair bit of effort in setting up and 
> running tests, but not as flexible at validating the generated output. 
> The tcl/expect framework, is pretty good at that, so perhaps a 
> combination could be used, TCS to run these tests and dejagnu to do the 
> compare of the output.

The biggest problem with output compare is expect merges the 
stdout/stderr streams, while TCS puts all stderr output first, followed 
by all stdout output.  I've already dealt with this is a great extent in 
the current TCS/Tcl, but I'm sure there are still issues.

-John

RE: [Firebird-test] What to do with TCS

[Leyne, S. <sl...@at...>]

John,

> 
> But we need to decide what to do with TCS first...
> 

Personally, right at the moment given the timelines that we need to
follow -- my vote is to modify TCS to use flat/XML files (so at least we
get rid of the DB dependency) and _perhaps_ update the code so that it
can compile under c++ with autoconf (just like the v2 engine work).

Once we have v1 out the door, we can start a broader review of the
various approaches.


Sean

Re: [Firebird-test] What to do with TCS

[Mark O'D. <mar...@lu...>]

John Bellardo wrote:

>
>
> But we need to decide what to do with TCS first...

Well Im all for giving it the flick, and extracting the tests into CVS 
as flat files :-).

But I'd really like to hear what Paul Reeves has to say (and probably 
Frank), since they are the ones that have really done some worked with 
it.  And I suspect they will have found some merrit in keeping it.

One think I have been thinking off, is a modified tcs program that works 
from flat files could still be run from dejagnu, with perhaps an option 
to use dejagnu to filter and compare the output.

TCS seems to me to represent a fair bit of effort in setting up and 
running tests, but not as flexible at validating the generated output. 
 The tcl/expect framework, is pretty good at that, so perhaps a 
combination could be used, TCS to run these tests and dejagnu to do the 
compare of the output.


Cheers

Mark

-- 
Your database needs YOU!
http://firebird.sourceforge.net 
(bugger, quick before Sean finds out -^H^H^H^H^H^Hwww.firebirdsql.org 
:-).

GPL: free software today - and still free tomorrow

Re: [Firebird-test] What to do with TCS

[John B. <bel...@cs...>]

On Wednesday, August 8, 2001, at 11:35 PM, Mark O'Donohue wrote:

> John Bellardo wrote:
>
>>
>> There are a number of other advantages, including the use of flat 
>> files and the fact dejagnu is a fairly standard testing tool.  Mark 
>> was saying the Postgres and mySQL test suites are written in dejagnu.  
>> It may be possible to use some of them with FB.
>
>
> Sorry I misled you a bit there, rather that downloading cygwin for 
> windows (which did run dejagnu and my test expect program) also by 
> default dowloaded everything including postgres.
>
> I haven't looked at the postgres and mysql suites in detail but I don't 
> think they are written in dejagnu, (but they are both unix/cygwin shell 
> script based).

Not a big deal.

>
> I also was suggesting we may be able to use tcl/expect to run some of 
> the above (crashme from mysql comes to mind) and the nist test suite.

The more tests we can get the better, regardless of there source (as 
long as they are legal).

But we need to decide what to do with TCS first...

-John

Re: [Firebird-test] What to do with TCS

[Mark O'D. <mar...@lu...>]

John Bellardo wrote:

> Hi,
>
> I've seen two ideas:
>
> 1.  A modified TCS/c that uses flat files.
>
> 2.  Dejagnu (Tcl/expect based).

I believe that Paul Reeves is working on a modified tcs routine, but I 
think having both in some form is probably going to be a good thing.

A modified TCS that is easy to install and allows a user to develop a 
few tests may be handly to get more people involved in running test cases.

For the serious regression testing the dejagnu thing I think will be better.

It can be used for generating new tests and co-opting some of the other 
non TCS test suites (nist etc).  These can be run on all platforms, but 
require more installation than just a download of a "c" exe and a few 
config files.

Im still keen on a thrasher thing, which I suspect will need to be a c 
program (and I aint' looked at your's yet :-).

Cheers


Mark

Re: [Firebird-test] What to do with TCS

[Mark O'D. <mar...@lu...>]

John Bellardo wrote:

>
> There are a number of other advantages, including the use of flat 
> files and the fact dejagnu is a fairly standard testing tool.  Mark 
> was saying the Postgres and mySQL test suites are written in dejagnu.  
> It may be possible to use some of them with FB.


Sorry I misled you a bit there, rather that downloading cygwin for 
windows (which did run dejagnu and my test expect program) also by 
default dowloaded everything including postgres.

I haven't looked at the postgres and mysql suites in detail but I don't 
think they are written in dejagnu, (but they are both unix/cygwin shell 
script based).

I also was suggesting we may be able to use tcl/expect to run some of 
the above (crashme from mysql comes to mind) and the nist test suite.

Cheers

Mark

[Firebird-test] What to do with TCS

[John B. <bel...@cs...>]

Hi,
	I've been toying with the problem of replacing TCS while still 
being able to use all the existing tests.  Dejagnu was proposed as a 
possible new testing system, but that doesn't address TCS 
compatibility.  So I spent some time re-implementing TCS in Tcl (the 
language dejagnu uses).  My current implementation runs off the flat 
files generated from Frank S-G's TCS dump scripts, all flat files.

TCS/Tcl appears to function correctly with the C_SQL_JOIN, C_SQL_PRED, 
CF_ISQL, and QA_PROCS_ISQL series.  There are still a number of issues 
that need to be ironed out, but I figured I tell everyone what I've been 
up to.  This brings me to the issue at hand, what system do we want to 
use for our regression testing?

I've seen two ideas:

1.  A modified TCS/c that uses flat files.

2.  Dejagnu (Tcl/expect based).

The TCS/Tcl work I've done shows it is feasible to use (2), and that is 
what I'm advocating.  Tcl as a testing language gives us precise control 
over the interactions between the programs being tested, something that 
is very difficult in c.  For example Mark wrote a small script to test 
limit.  The script is able to count the number of rows returned from a 
query.  Using these sort of techniques is much easier than trying to 
implement them in c or manually inspecting the result of the queries to 
ensure the diff file is correct.

There are a number of other advantages, including the use of flat files 
and the fact dejagnu is a fairly standard testing tool.  Mark was saying 
the Postgres and mySQL test suites are written in dejagnu.  It may be 
possible to use some of them with FB.

What do other people think?  It would be nice to decide on a direction 
so we can make more focused progress.

Thanks.

-John