firebird-test

[Firebird-test] results of qmtest run on linux fb2.0.4

[Mark O'D. <mar...@gm...>]

So it seems for fb 2.0.4 (linux) we have 402 (358 + 44 ) test cases, is 
that correct or are there more somewhere else?

$qmtest -C context.txt run functional

--- STATISTICS 
---------------------------------------------------------------

      358        tests total
        1 (  0%) tests ERROR
        2 (  1%) tests FAIL
      355 ( 99%) tests PASS



$qmtest -C context.txt run bugs


--- STATISTICS 
---------------------------------------------------------------

       44        tests total
        2 (  5%) tests ERROR
        2 (  5%) tests FAIL
       40 ( 91%) tests PASS


I re-ran one of the FAILS and it worked the second time (database 
already existed was the cause).  Is there an official list of results I 
can check to see how these compare?


Cheers - Mark


Below are the ones that failed.



--- TESTS THAT DID NOT PASS 
--------------------------------------------------

   functional.arno.optimizer.opt_multi_index_selection_06: FAIL
     Expected standard output from ISQL does not match actual output.

   functional.arno.optimizer.opt_single_index_selection_11: FAIL
     Expected standard output from ISQL does not match actual output.

   functional.intfunc.cast.cast_01               : ERROR
     Exception raised while creating database.



--- TESTS THAT DID NOT PASS 
--------------------------------------------------

   bugs.core_1006                                : ERROR
     Backup file core1006.fbk does not exist.

   bugs.core_1063                                : ERROR
     Backup file core1063.fbk does not exist.

   bugs.core_1331                                : FAIL
     Expected STDERR stream received from ISQL.

   bugs.core_959                                 : FAIL
     Expected standard output from python does not match actual output.

Re: [Firebird-test] results of qmtest run on linux fb2.0.4

[Philippe M. <mak...@fi...>]

Mark O'Donohue  [08-12-13 15.10] :
> So it seems for fb 2.0.4 (linux) we have 402 (358 + 44 ) test cases, is 
> that correct or are there more somewhere else?
> 
Classic or SuperServer ?

> 
>    functional.arno.optimizer.opt_multi_index_selection_06: FAIL
>      Expected standard output from ISQL does not match actual output.
> 
>    functional.arno.optimizer.opt_single_index_selection_11: FAIL
>      Expected standard output from ISQL does not match actual output.
> 
These two are true bugs, so it is normal that they fail

>    functional.intfunc.cast.cast_01               : ERROR
>      Exception raised while creating database.
this one should pass
before running test, check that you don't have databases in your "database_location" directory

> 
>    bugs.core_1006                                : ERROR
>      Backup file core1006.fbk does not exist.
> 
>    bugs.core_1063                                : ERROR
>      Backup file core1063.fbk does not exist.
> 
these two should pass
check your "backup_location" directory and check also that firebird user can read files

>    bugs.core_1331                                : FAIL
>      Expected STDERR stream received from ISQL.
> 
this one "officialy" fail, it is a charset problem with qmtest

>    bugs.core_959                                 : FAIL
>      Expected standard output from python does not match actual output.
> 
This one I allways have trouble to run it with Superserver, but not with Classic
but it is a qmtest problem

so not far from the "official" results

there is no public place where you could find results, Pavel and I run tests, and after
that we create a report with all results files using the
qmreport python script you can find in the svn

Re: [Firebird-test] results of qmtest run on linux fb2.0.4

[Mark O'D. <mar...@gm...>]

Hi Phillippe

On 14/12/08 02:46, Philippe Makowski wrote:
> Mark O'Donohue  [08-12-13 15.10] :
>> So it seems for fb 2.0.4 (linux) we have 402 (358 + 44 ) test cases, is
>> that correct or are there more somewhere else?
>>
> Classic or SuperServer ?
>

That was super.

Thanks for those, its nice to be getting similar answers :-)

I reran functional.intfunc.cast.cast_01  and it worked, the ERROR was 
that the database already existed, but knowing I am in the ballpark I've 
moved on to trying to run against my 2.5 build.

>
> there is no public place where you could find results, Pavel and I run tests, and after
> that we create a report with all results files using the
> qmreport python script you can find in the svn

Once the report is created, is it posted somewhere, or put on the website?


Also with the bugs test suite, we seem to have a variety of tests from 
core_86.qmt ... core_1885.qmt,  but tracker seems to be up to bug 
core-2238.   There seems to be a mix of information available in 
tracker, most of it helpful for building the test case.

But I am curious as to who is responsible for building the test case for 
each bug (the usual process is to dob in either the reporter or the 
developer as the bunny)?


Cheers - Mark

Re: [Firebird-test] results of qmtest run on linux fb2.0.4

[Philippe M. <mak...@fi...>]

Mark,
> I reran functional.intfunc.cast.cast_01  and it worked, the ERROR was 
> that the database already existed, but knowing I am in the ballpark I've 
> moved on to trying to run against my 2.5 build.
you will have a lot of fail tests, since 2.5 change error codes

> Once the report is created, is it posted somewhere, or put on the website?
> 
no
Pavel have all, and send them to dev if need.

> But I am curious as to who is responsible for building the test case for 
> each bug (the usual process is to dob in either the reporter or the 
> developer as the bunny)?
> 
dev resolve bug, Q/A team test if it is really solved, close the bug and if possible
create the qmtest test.
So, Pavel and I, with sometime help from french students (and recently Roman Simakov from
Red Soft committed tests for FK) are the one that do the job.

Re: [Firebird-test] results of qmtest run on linux fb2.0.4

[Roman S. <rom...@re...>]

Philippe Makowski wrote:
> dev resolve bug, Q/A team test if it is really solved, close the bug and if possible
> create the qmtest test.
> So, Pavel and I, with sometime help from french students (and recently Roman Simakov from
> Red Soft committed tests for FK) are the one that do the job.
>
>   
Hi Philippe, All!

A small correction ;) I just committed the tests since i have necessary 
rights but Vyacheslav Ozherelyev (vya...@re...) is 
a real developer of them :)
He can answer you about they. Red Soft has many changes of source code 
and will provide a set of patches for firebird v3.0. Some of them 
already discussed and probably will be merged into cvs tree after 
branching 2.5. As well as core patches Red Soft provides a set of 
qm-tests for new functionality. Vyacheslav is responsible for it and he 
is ready to tell you about exists tests.

Roman Simakov

Re: [Firebird-test] results of qmtest run on linux fb2.0.4

[Philippe M. <mak...@fi...>]

Roman Simakov  [08-12-20 11.20] :
> A small correction ;) I just committed the tests since i have necessary 
> rights but Vyacheslav Ozherelyev (vya...@re...) is 
> a real developer of them :)
yes sorry

> He can answer you about they. Red Soft has many changes of source code 
> and will provide a set of patches for firebird v3.0. Some of them 
> already discussed and probably will be merged into cvs tree after 
> branching 2.5. As well as core patches Red Soft provides a set of 
> qm-tests for new functionality. Vyacheslav is responsible for it and he 
> is ready to tell you about exists tests.
> 
Are they all for 3.0 ? or is there some for 2.5 ?

We will have two French students that will help us for Q/A, so we have to tell them what
kind of tests we need.

Re: [Firebird-test] results of qmtest run on linux fb2.0.4

[Vyacheslav O. <vya...@re...>]

Philippe Makowski wrote:
> Roman Simakov  [08-12-20 11.20] :
>   
>> A small correction ;) I just committed the tests since i have necessary 
>> rights but Vyacheslav Ozherelyev (vya...@re...) is 
>> a real developer of them :)
>>     
> yes sorry
>
>   
>> He can answer you about they. Red Soft has many changes of source code 
>> and will provide a set of patches for firebird v3.0. Some of them 
>> already discussed and probably will be merged into cvs tree after 
>> branching 2.5. As well as core patches Red Soft provides a set of 
>> qm-tests for new functionality. Vyacheslav is responsible for it and he 
>> is ready to tell you about exists tests.
>>
>>     
> Are they all for 3.0 ? or is there some for 2.5 ?
>   
Yes, all for 3.0
> We will have two French students that will help us for Q/A, so we have to tell them what
> kind of tests we need.
>
>   

Regards
Vyacheslav Ozherelyev