firebird-test — Firebird engine testing, test flight/suite discussion/information

RE: [Firebird-test] WBT: Code Audit

[Pavel C. <pc...@us...>]

Sean,

On 3 Jul 2002 at 22:35, Leyne, Sean wrote:

> I wrote;
> 
> > > get deoxygen involved and stress the importance of adding internal 
> > > documentation.
> > 
> > I completely agree!
> > 
> > In fact, if we get agreement that this is the project 
> > standard and the extent that we want to proceed initially, 
> > then I would be prepared to start updating the source to 
> > provide the documentation.
> 
> I'd like to withdraw my previous endorsement.
> 
> I've just read the installation instructions!!!  What is it with
> people/developers?  They all assume that we are C/C++ nuts and have
> compilers/Cygwin at our finger tips!  Why can't they simply make Win32
> binaries available.

Hold the horses ! I don't know what instructions you've read, but I've 
just downloaded and the installed Win32 version of doxygen (It's a 
selfcontaining executable with size around 4MB). It doesn't need any 
compiler or cygwin to run on Windows, and from a brief look it seems to 
be a quite capable tool. So don't worry and go for it!

Best regards
Pavel Cisar
http://www.ibphoenix.com
For all your upto date Firebird and
InterBase information

RE: [Firebird-test] BBT: Automated QA and TCS

[Pavel C. <pc...@us...>]

Sean,

On 3 Jul 2002 at 22:08, Leyne, Sean wrote:

> Just as a reminder of a historical posting, Reed Mideke wrote:
> 
> > Emm, actually, embedded GDML (i.e. GPRE) is the gold standard.
> > isql only lets you test DSQL. 
> 
> Accordingly, perhaps we need to balance the use of GPRE based tests and
> DSQL/isql based test.
>
> Ideally, the GPRE test should cover the very basic elements of the
> database/engine and those features which can't be tested any other
> way. Once the basics have been tested then other tools (like isql) can
> be used to perform the reminder (majority) of the tests.  In the long
> run, these basic tests should change less frequently than the need to
> create new test to address DSQL issues. 

I agree with you that we'll need (at least some) ESQL tests, because not 
all functionality is exposed by DSQL, and significant engine parts and 
tools are written in it. We should tests these "bits" that cannot be 
tested by other ways.

> To address the deployment issues, installing/configuring a C/C++
> compiler, we could build binary packages which could be installed by
> users/testers (this is in addition to the test package -- no CVS access
> should be required).  Thus, only 1 person would need to compile and
> build the binary package, and another to build/post the package of
> testing scripts. 
>
> Then, the thousands of users can contribute to the overall testing
> process. 

I'm not sure if 1000+1 users running _exactly_ the same tests would 
assure better quality than 1-3 (per platform). I'd prefer if those 1000+1 
users would test FB in real-world applications.

Best regards
Pavel Cisar
http://www.ibphoenix.com
For all your upto date Firebird and
InterBase information

RE: [Firebird-test] WBT: Code Audit

[Leyne, S. <sl...@at...>]

I wrote;

> > get deoxygen involved and stress the importance of adding internal=20
> > documentation.
>=20
> I completely agree!
>=20
> In fact, if we get agreement that this is the project=20
> standard and the extent that we want to proceed initially,=20
> then I would be prepared to start updating the source to=20
> provide the documentation.

I'd like to withdraw my previous endorsement.

I've just read the installation instructions!!!  What is it with
people/developers?  They all assume that we are C/C++ nuts and have
compilers/Cygwin at our finger tips!  Why can't they simply make Win32
binaries available.


Sean

RE: [Firebird-test] WBT: Code Audit

[Leyne, S. <sl...@at...>]

> > 3) What do you think / recommend re. Code audit ?
>=20
> get deoxygen involved and stress the importance of adding internal=20
> documentation.

I completely agree!

In fact, if we get agreement that this is the project standard and the
extent that we want to proceed initially, then I would be prepared to
start updating the source to provide the documentation.


Sean

RE: [Firebird-test] BBT: Automated QA and TCS

[Leyne, S. <sl...@at...>]

Paul et al,

> From: Paul Reeves [mailto:pr...@ib...]

> Pavel Cisar wrote:
>=20
> > Tests mostly use ESQL for access to the database,=20
> > instead of DSQL which is prevailing approach today.=20
>=20
> How would you see us testing DSQL? I'd trust the results of=20
> an ESQL test=20
> over a DSQL test anyday. Writing good (I mean correct) DSQL=20
> tests is not=20
> easy. At least gpre knows what it is doing.
>=20
> This is not a reason to ignore testing DSQL, especially as gpre=20
> generates undocumented api calls in some cases. But having played a=20
> little with writing gpre scripts I think the learning curve is more=20
> shallow. Test writers can concentrate on writing code that will=20
> accomplish a test. Writing DSQL will turn into a test of the=20
> programmers=20
> C/C++ abilities - at least in the beginning.
>=20
> I think one of the goals of the QA group is to create an environment=20
> where tests can be created quickly and easily by many=20
> developers. ESQL makes this a lot easier.


Just as a reminder of a historical posting, Reed Mideke wrote:

> > The current ISQL approach virtually eliminated these type of errors,
> > saving testers from hunting down these 'false positive' results.
> > Furthermore, when it comes to diagnosing anomolies in data=20
> results, ISQL
> > has been used as the "gold standard" for determining=20
> whether a problem
> > is engine or data access layer related.
> >=20
> Emm, actually, embedded GDML (i.e. GPRE) is the gold standard.
> isql only lets you test DSQL.=20

Accordingly, perhaps we need to balance the use of GPRE based tests and
DSQL/isql based test.

Ideally, the GPRE test should cover the very basic elements of the
database/engine and those features which can't be tested any other way.
Once the basics have been tested then other tools (like isql) can be
used to perform the reminder (majority) of the tests.  In the long run,
these basic tests should change less frequently than the need to create
new test to address DSQL issues.

To address the deployment issues, installing/configuring a C/C++
compiler, we could build binary packages which could be installed by
users/testers (this is in addition to the test package -- no CVS access
should be required).  Thus, only 1 person would need to compile and
build the binary package, and another to build/post the package of
testing scripts.=20

Then, the thousands of users can contribute to the overall testing
process.

...

> My main objection (in fact my only objection) to QMTest is=20
> that it requires Python. It is another thing to install and
> configure and it quite probably requires learning a bit of
> python, too. That said, I'm not stuck on this.

I also share this opinion.  But at the same time we need a solution
which:

1) runs well on Windows (I lot of our users and potential new
developers/testers use Windows)

2) runs on all platforms without requiring too many supporting tools

3) provides a good/meaningful display of the test results and error
details

4) stores test scripts in individual text/xml files

5) can be installed with a straight-forward set of installation notes
that even I can follow (in other words, idiot-proof!).

It seems that QMTest meets these requirements, but I'm open to others.


Sean

Re: [Firebird-test] BBT: Automated QA and TCS

[John B. <bel...@cs...>]

Pavel,

On Wednesday, July 3, 2002, at 12:49 PM, Pavel Cisar wrote:

> Hi,
>
> On 3 Jul 2002 at 10:09, John Bellardo wrote:
>
>> One of the things that would help here is the ability to take generic
>> sql scripts and run them through all the different interfaces (esql,
>> dsql, etc).  This would allow us to drive common aspects of our
>> different interfaces from the same test script.  Write once and test
>> everywhere.  Obviously it will only work for the common subset of
>> interface functionality (dsql doesn't have blob access, for example).
>> But a lot of the tests in TCS are glorified SQL scripts already and
>> would benefit from this.
>
> This would be really nice. But pardon my ignorance (I'm not an ESQL
> guru), how we can run arbitrary SQL scripts via ESQL ? I don't know that
> this is possible directly, so would need some transformation tool (jet
> another kind of GPRE) ?

There would need to be a small transformation tool, but it shouldn't be 
too complex.  It would take as input the SQL and spit out a .e (or .epp) 
file.

> [...]

-John

Re: [Firebird-test] WBT: Code Audit

[John B. <bel...@cs...>]

On Wednesday, July 3, 2002, at 12:49 PM, Pavel Cisar wrote:

> Hi John,
>
> On 3 Jul 2002 at 9:57, John Bellardo wrote:
>
>> get deoxygen involved and stress the importance of adding internal
>> documentation.
>
> Do you have an URL to deoxygen ?

http://www.stack.nl/~dimitri/doxygen/
I guess I messed up the spelling before.

> As I'm not an C developer, I don't know
> it (but I expect that it's some sort of JavaDoc/PasDoc, right ?)

correct.

-John

Re: [Firebird-test] WBT: Code Audit

[Pavel C. <pc...@us...>]

Hi John,

On 3 Jul 2002 at 9:57, John Bellardo wrote:

> get deoxygen involved and stress the importance of adding internal 
> documentation.

Do you have an URL to deoxygen ? As I'm not an C developer, I don't know 
it (but I expect that it's some sort of JavaDoc/PasDoc, right ?)

Best regards
Pavel Cisar
http://www.ibphoenix.com
For all your upto date Firebird and
InterBase information

Re: [Firebird-test] BBT: Automated QA and TCS

[Pavel C. <pc...@us...>]

Hi,

On 3 Jul 2002 at 10:09, John Bellardo wrote:

> One of the things that would help here is the ability to take generic 
> sql scripts and run them through all the different interfaces (esql, 
> dsql, etc).  This would allow us to drive common aspects of our 
> different interfaces from the same test script.  Write once and test 
> everywhere.  Obviously it will only work for the common subset of 
> interface functionality (dsql doesn't have blob access, for example).  
> But a lot of the tests in TCS are glorified SQL scripts already and 
> would benefit from this.

This would be really nice. But pardon my ignorance (I'm not an ESQL 
guru), how we can run arbitrary SQL scripts via ESQL ? I don't know that 
this is possible directly, so would need some transformation tool (jet 
another kind of GPRE) ?
 
> At one point I had ported most, if not all, of TCS to dejagnu.  So I 
> guess that is also a consideration.  But the interface on that is 
> admittedly poor compared to some of the GUI-based test harnesses.

My main objection to dejagnu/expect solution is that it IMHO really 
doesn't solve the main problem - How tests are written. Actually, TCS 
it's not so bad once you make it running, but tests written in mixture of 
shell script, arcane TCS commands, environment variables (both OS and 
TCS), C/C++, ESQL etc. with use of various external tools (thanks god all 
standard, free or part of TCS suite) and all packed in single text file 
(sorry, I mean BLOB in test database ,-) is really a nightmare. Of 
course, dejagnu/expect can shallow it a bit but at expense that we have 
to rewrite all tests. I we have to write all tests from scratch, I'd like 
try to find more "test writer friendly" solution first. But dejagnu is 
still in play as backup solution.
 
> I think the barrier to entry for the test suite can be higher than that 
> of the engine.  For example, I wouldn't have a problem with the would be 
> tester had to install one or two additional software packages to test 
> (5-10 would be excessive) as long as the process the clearly documented, 
> uses only free tools (of course), and only takes a few minutes aside 
> from download time.  Our testers should not have to learn any additional 
> programming language just to run the tests.  It is a reasonable 
> expectation that they need to know some language to write tests.

I agree, but easy to setup & operable test harness would be a plus. 
Actually, it's not supposed that test harness should be used by more than 
few people -> "platform keepers/packagers and QA specialists" (of course, 
it would be nice if anyone would be able to verify our results, but it's 
not required). But we do not have enough good tests (and we speak about 
thousands tests here) and we need them quickly. That is the main problem 
we face here, regardless of concrete test harness we close with. Since 
Firebird project inception we learned that skilled C/C++ developers are 
scarce to be wasted on something like test cases, even if we'll convince 
them to write them. But there are many Delphi, Java, PHP and SQL 
developers flying around Firebird project that are not C/C++ developers 
and thus disqualified for core engine development, waiting for an 
opportunity to do something useful. We should come with a solution that 
allow these developers to write some tests in their spare time. Not much 
work, not much learning, not much intellectual load. It's IMO the only 
way we can get tests we need and "in time". For comparison, if I'd be 
paid to work on test cases full time, and I'll write (and verify) ten 
tests a day, it will take almost two months for me just to replace TCS 
tests. And we'll need ten times more tests just to match Borland's 
"certification". We need an approach that either allow me to write more 
tests a day (not feasible, new test must be _designed_, written and 
tested and it's a lot of handwork on itself), or allow us to turn in more 
people (usually new, not already involved in Firebird developments). I 
can't see any other way than lowering the entry barrier for test writers. 
But how ? Well, declarative tests can work for some tested areas (common 
SQL, PSQL) but not for all (events, ISQL etc.). But how "declarative 
test" should be "declared" ? How other test should be build ? That is the 
question, dear Horatio :) I've some ideas, but I'd greatly appreciate any 
suggestion about how to make test writing more sexy for anyone.

Best regards
Pavel Cisar
http://www.ibphoenix.com
For all your upto date Firebird and
InterBase information

Re: [Firebird-test] BBT: Automated QA and TCS

[Pavel C. <pc...@us...>]

On 3 Jul 2002 at 15:31, Paul Reeves wrote:

> How would you see us testing DSQL? I'd trust the results of an ESQL test 
> over a DSQL test anyday. Writing good (I mean correct) DSQL tests is not 
> easy. At least gpre knows what it is doing.

AFAIK any db access layer that's in use today works over DSQL (IBO, IBX, 
ODBC, dbExpress, kinterbasdb, JDBC? etc.). So use of DSQL means that we 
can simply use one such library (in case of QMTest it's kinterbasdb). 
 
> This is not a reason to ignore testing DSQL, especially as gpre 
> generates undocumented api calls in some cases. But having played a 
> little with writing gpre scripts I think the learning curve is more 
> shallow. Test writers can concentrate on writing code that will 
> accomplish a test. Writing DSQL will turn into a test of the programmers 
> C/C++ abilities - at least in the beginning.

I think the opposite (in favour of DSQL), because with ESQL we have to 
use a C/C++ compiler, GPRE and other tools just to compile test. With 
DSQL, the db access layer could be part of Test harness, and test cases 
can use declarative approach instead procedural. Of course, some test are 
better written in ESQL/GPRE (like tests for events), but core SQL 
compliance tests could be pure declarations: sql command -> expected 
results.

> Have you seen QAT yet? (http://qat.sourceforge.com). It is another OS 
> test harness, this time written in Java. I haven't played with it 
> seriously but the fact that it is in Java makes it more interesting to me.

Java is also nice, as one from top priority requirements for test harness 
is portability, but it depends on working Java FB connectivity. Of 
course, we can use it at least for jca-jdbc driver tests. I'll take a 
look at QAT.
 
> My main objection (in fact my only objection) to QMTest is that it 
> requires Python. It is another thing to install and configure and it 
> quite probably requires learning a bit of python, too. That said, I'm 
> not stuck on this. A harness that is easily configurable and allows easy 
> extension addition to the test 'database' is the most important thing. 
> We need to reduce the barriers to test writing as much as possible.

Python is easy to setup and QMTest as well (just install them and you're 
ready). Many test cases doesn't really need any Python knowledge as I'd 
like provide support for declarative approach for test writers where 
possible. This is also one from main topics for discussion here: How 
common SQL compliance test should be written and what support is expected 
from TCS side to build them quickly without special knowledge and with 
minimum coding.

Best regards
Pavel Cisar
http://www.ibphoenix.com
For all your upto date Firebird and
InterBase information

Re: [Firebird-test] BBT: Automated QA and TCS

[John B. <bel...@cs...>]

Hi,

On Wednesday, July 3, 2002, at 06:31 AM, Paul Reeves wrote:

> Pavel Cisar wrote:
>
>
>> Tests mostly use ESQL for access to the database, instead of DSQL 
>> which is prevailing approach today.
>
> How would you see us testing DSQL? I'd trust the results of an ESQL 
> test over a DSQL test anyday. Writing good (I mean correct) DSQL tests 
> is not easy. At least gpre knows what it is doing.

One of the things that would help here is the ability to take generic 
sql scripts and run them through all the different interfaces (esql, 
dsql, etc).  This would allow us to drive common aspects of our 
different interfaces from the same test script.  Write once and test 
everywhere.  Obviously it will only work for the common subset of 
interface functionality (dsql doesn't have blob access, for example).  
But a lot of the tests in TCS are glorified SQL scripts already and 
would benefit from this.

>
> [...]
>
>> We'll need more friendly TCS. Open source QMTest 
>> (www.codesourcery.com) is an option, but you can suggest other tools 
>> (including home-grown) as well.
>
>
> Have you seen QAT yet? (http://qat.sourceforge.com). It is another OS 
> test harness, this time written in Java. I haven't played with it 
> seriously but the fact that it is in Java makes it more interesting to 
> me.
>

At one point I had ported most, if not all, of TCS to dejagnu.  So I 
guess that is also a consideration.  But the interface on that is 
admittedly poor compared to some of the GUI-based test harnesses.

> My main objection (in fact my only objection) to QMTest is that it 
> requires Python. It is another thing to install and configure and it 
> quite probably requires learning a bit of python, too. That said, I'm 
> not stuck on this. A harness that is easily configurable and allows 
> easy extension addition to the test 'database' is the most important 
> thing. We need to reduce the barriers to test writing as much as 
> possible.

I think the barrier to entry for the test suite can be higher than that 
of the engine.  For example, I wouldn't have a problem with the would be 
tester had to install one or two additional software packages to test 
(5-10 would be excessive) as long as the process the clearly documented, 
uses only free tools (of course), and only takes a few minutes aside 
from download time.  Our testers should not have to learn any additional 
programming language just to run the tests.  It is a reasonable 
expectation that they need to know some language to write tests.

-John

Re: [Firebird-test] WBT: Code Audit

[John B. <bel...@cs...>]

Hi,

On Wednesday, July 3, 2002, at 04:04 AM, Pavel Cisar wrote:

> Hi all,
>
> Code audit is just a form of Peer Review.[...]
>
> Implementation:
> Every source file in _Firebird2_ module (29MB) would be reviewed. It
> would be nice if these files would be also _documented_. That mean both,
> in-line comments and separate FB internals document, where major data
> structures and subsystems would be documented, including their mutual
> relations.
>
> Current status:
> This technique was not used (at all AFAIK), except as a by-product of
> learning.
>
> Questions for you:
> 1) Do you think that we should do Code review as is explained here ?

Not exactly.  I think we don't have the resources to do a full "stop 
everything" code review.  We should do them incrementally as the various 
engine subsystems are examined in detail.  For example, I just spent a 
fair amount of time in the intl module and could have done a review then 
without too much additional burden.

>
>
> 2) If yes, what should be the primary purpose: code correctness or
> documentation ?

Right now, documentation.  I would like to see deoxygen become a 
standard part of the FB build process, and engine internal information 
placed in deoxygen comments in the code.  Once we have determined how 
the code is functioning now we can talk about how it is supposed to 
function.  That will help us identify bugs.  For the time being the only 
bugs we can really catch (in most subsystems) are small typos.

>
> 3) What do you think / recommend re. Code audit ?

get deoxygen involved and stress the importance of adding internal 
documentation.

-John

Fwd: Re: [Firebird-test] WBT: Development test cases [davidjencks@directvinternet.com]

[David J. <dav...@di...>]

Those reply to headers got me again.

On 2002.07.03 08:40:21 -0400 David Jencks wrote:
My experience is that, whenever I write them, my software only does what I
think if there is a unit test.  Although the tests are not terribly
complete, I think the client-java driver would not have gotten very far
without the junit test framework and tests.  My experience is that after
the testing framework is set up it is very easy to write new tests, and
gets to be almost fun quickly;-)

There appear to be several c++ unit test frameworks now:

http://www.xprogramming.com/software.htm

I can't really judge which one would be most appropriate, but based on my
java experience we will get great rewards if someone takes the time to set
up the infrastructure.

Thanks
david jencks

On 2002.07.03 07:04:13 -0400 Pavel Cisar wrote:
> Hi all,
> 
> Developers often use home-made test applications to test basic 
> functionality of parts they're working on. They are not always developed 
> nor complete, but if they are developed, they are not deleted afterwards 
> and could be reused in test system or by QA group. They don't need to be 
> so extreme like in eXtreme programming, and developed *before* any real 
> development take place (although XP approach would be nice :), but *any* 
> such application has its value.  
> 
> Pro's and Con's: 
> 
> This technique impose significant burden to core developers, if they're 
> required to write test cases along their code. Of course, these tests 
> could be written by others, but this task require intimate knowledge at 
> least tested piece of source code. Tests must be written in certain 
> standard way, to assure seamless integration into automatic test system. 
> Test applications that would test correctness of various low-level 
> functions or modules/subsystems can quickly uncover bugs that are 
> otherwise hard to catch at higher level, especially those that are 
> related to parameter values and execution flow handling (if/else/switch).
> 
> From my experience, around 50% of all bugs are related to bad 
> if/else/switch handling (undetected/badly handled values etc.). These 
> tests can also work as a specification for produced code.   
> 
> Implementation:
> 
> We can initiate a sub-project in Firebird to build such test programs for
> 
> selected libraries and separate subsystems (like memory management, 
> optimizer, parser etc.)  and/or declare a policy for core developers 
> about when and how write such test cases. The basic prerequisite to start
> 
> it successfully is a good source code documentation (file dependencies, 
> list of routines and symbols etc.) to allow a seasoned C/C++ developers 
> to get on track quickly. We have at (least basic) such documentation, and
> 
> for rest it's exactly where GLOBAL tagging systems would come handy.  
> 
> Current status:
> 
> We don't have anything like this for Firebird right now.
> 
> Questions for you:
> 1) Do you think that we should use low-level test cases ? 
> 
> 
> 2) If yes, what parts of core engine do you suggest to be a subject of 
> this testing ?
> 
> 
> 3) If yes, what procedure and/or toolkit do you recommend ?
> 
> 
> Your comments would be greatly appreciated.
> Best regards
> Pavel Cisar
> http://www.ibphoenix.com
> For all your upto date Firebird and
> InterBase information
> 
> 
> 
> -------------------------------------------------------
> This sf.net email is sponsored by:ThinkGeek
> No, I will not fix your computer.
> http://thinkgeek.com/sf
> _______________________________________________
> Firebird-test mailing list
> Fir...@li...
> https://lists.sourceforge.net/lists/listinfo/firebird-test
> 
> 

Re: [Firebird-test] BBT: Automated QA and TCS

[Paul R. <pr...@ib...>]

Pavel Cisar wrote:


> Tests mostly use ESQL for access to the database, 
> instead of DSQL which is prevailing approach today. 

How would you see us testing DSQL? I'd trust the results of an ESQL test 
over a DSQL test anyday. Writing good (I mean correct) DSQL tests is not 
easy. At least gpre knows what it is doing.

This is not a reason to ignore testing DSQL, especially as gpre 
generates undocumented api calls in some cases. But having played a 
little with writing gpre scripts I think the learning curve is more 
shallow. Test writers can concentrate on writing code that will 
accomplish a test. Writing DSQL will turn into a test of the programmers 
C/C++ abilities - at least in the beginning.

I think one of the goals of the QA group is to create an environment 
where tests can be created quickly and easily by many developers. ESQL 
makes this a lot easier.


> We'll need more friendly TCS. Open source QMTest (www.codesourcery.com) 
> is an option, but you can suggest other tools (including home-grown) as 
> well.


Have you seen QAT yet? (http://qat.sourceforge.com). It is another OS 
test harness, this time written in Java. I haven't played with it 
seriously but the fact that it is in Java makes it more interesting to me.

My main objection (in fact my only objection) to QMTest is that it 
requires Python. It is another thing to install and configure and it 
quite probably requires learning a bit of python, too. That said, I'm 
not stuck on this. A harness that is easily configurable and allows easy 
extension addition to the test 'database' is the most important thing. 
We need to reduce the barriers to test writing as much as possible.


Paul
-- 

Paul Reeves
http://www.ibphoenix.com
Supporting users of Firebird and InterBase

[Firebird-test] WBT: Design by contract

[Pavel C. <pc...@us...>]

Hi all,

In languages without direct DBC support it mean asserts for 
precondition, postcodition and invariants included in the code.

Pro's and Con's:

This technique is a variant of Development test cases, but included 
directly to the source code. Can be used in both ways, as a substitution 
or complementary to dev. test cases. It's not hard to utilize and usually 
doesn't impose a significant burden to core developers.

Implementation:

First, we have to verify asserts already present in source code and 
usability of DEBUG build on all supported platforms. We can initiate a 
sub-project in Firebird to include asserts into selected libraries and 
separate subsystems and/or declare a policy for core developers 
about when and how include asserts into source code.

Current status:

There're asserts already in the engine, so we just need to capitalize on 
them, i.e. use DEBUG builds in testing. But I worry that they are not 
always up-to-date/correct nor complete, so review would be needed.  

Questions for you:
1) Do you think that we should use asserts ? 


2) If yes, in what parts of core engine do you think that we should 
include asserts ?


3) If yes, what procedure do you recommend ?

4) If yes, what should we focus on: precondition, postcondition, 
invariants, some of these, all or other not listed here ?


Your comments would be greatly appreciated.
Best regards

Pavel Cisar
http://www.ibphoenix.com
For all your upto date Firebird and
InterBase information

[Firebird-test] WBT: Development test cases

[Pavel C. <pc...@us...>]

Hi all,

Developers often use home-made test applications to test basic 
functionality of parts they're working on. They are not always developed 
nor complete, but if they are developed, they are not deleted afterwards 
and could be reused in test system or by QA group. They don't need to be 
so extreme like in eXtreme programming, and developed *before* any real 
development take place (although XP approach would be nice :), but *any* 
such application has its value.  

Pro's and Con's: 

This technique impose significant burden to core developers, if they're 
required to write test cases along their code. Of course, these tests 
could be written by others, but this task require intimate knowledge at 
least tested piece of source code. Tests must be written in certain 
standard way, to assure seamless integration into automatic test system. 
Test applications that would test correctness of various low-level 
functions or modules/subsystems can quickly uncover bugs that are 
otherwise hard to catch at higher level, especially those that are 
related to parameter values and execution flow handling (if/else/switch). 
From my experience, around 50% of all bugs are related to bad 
if/else/switch handling (undetected/badly handled values etc.). These 
tests can also work as a specification for produced code.   

Implementation:

We can initiate a sub-project in Firebird to build such test programs for 
selected libraries and separate subsystems (like memory management, 
optimizer, parser etc.)  and/or declare a policy for core developers 
about when and how write such test cases. The basic prerequisite to start 
it successfully is a good source code documentation (file dependencies, 
list of routines and symbols etc.) to allow a seasoned C/C++ developers 
to get on track quickly. We have at (least basic) such documentation, and 
for rest it's exactly where GLOBAL tagging systems would come handy.  

Current status:

We don't have anything like this for Firebird right now.

Questions for you:
1) Do you think that we should use low-level test cases ? 


2) If yes, what parts of core engine do you suggest to be a subject of 
this testing ?


3) If yes, what procedure and/or toolkit do you recommend ?


Your comments would be greatly appreciated.
Best regards
Pavel Cisar
http://www.ibphoenix.com
For all your upto date Firebird and
InterBase information

[Firebird-test] WBT: Code Audit

[Pavel C. <pc...@us...>]

Hi all,

Code audit is just a form of Peer Review. But while peer review as 
described in separate message address only code changes and small parts 
of codebase that's being work on, Code audit extends the peer review to 
the whole codebase. 

Pro's and Con's:
This technique is not used regularly (except for critical systems) 
because is very expensive in terms of human labour and required 
knowledge. But we're in unusual situation, because we inherited aprox. 
35MB of source code, where most of it was not touched by Firebird 
developers. There are large portions of code that only few (if anyone at 
all) knows in detail, and this knowledge is mostly a by-product of 
learning, so these developers read it to just decipher what's it supposed 
to do, not to check it's correctness. Of course, some bugs were found 
that way (like famous politically correct security hole), but it's mostly 
an exception. So, with Code audit we can: 

1) Learn more about engine internals, and DOCUMENT IT !!! The famous 
secret "Interbase internals" document is a little bit outdated and not 
complete. It's a nice opportunity to finish it.

2) We may find some bugs 

3) Identify potential flaws in design (at least optimizer come to mind 
here)


Implementation:
Every source file in _Firebird2_ module (29MB) would be reviewed. It 
would be nice if these files would be also _documented_. That mean both, 
in-line comments and separate FB internals document, where major data 
structures and subsystems would be documented, including their mutual 
relations.

Current status:
This technique was not used (at all AFAIK), except as a by-product of 
learning.

Questions for you:
1) Do you think that we should do Code review as is explained here ? 


2) If yes, what should be the primary purpose: code correctness or 
documentation ? 


3) What do you think / recommend re. Code audit ?


Your comments would be greatly appreciated.
Best regards
Pavel Cisar
http://www.ibphoenix.com
For all your upto date Firebird and
InterBase information

[Firebird-test] BBT: Automated QA and TCS

[Pavel C. <pc...@us...>]

Hi all,

First, a small recap about Black box testing:

<recap>
The purpose is to verify correctness and limits of final product. That 
mean:  

1) Correct input produce correct result.
2) Incorrect input produce correct error result.
3) Product reacts correctly to limit conditions and values (memory, 
disk space, file size, values at behaviour-switching boundary etc.)
4) Product reacts correctly to unexpected conditions (power failure, 
low-memory, I/O error)
5) Consistency. Test results from 1)-4) are consistent in time and 
with different order of execution. This also means concurrency tests.
6) Usability (performance, ergonomy, documentation)

If possible, automated test systems are used to do 1),2),3) and 5) 
tests. 4) and 6) usually depends completely on manual labour. All 
bugs and issues detected by QA department are tracked in some 
sort of Problem resolution system. This system is a crucial part of 
whole QA/development cycle, and any implemented QA procedure 
can't work successfully for long time without it. 

</recap>

Current status: 
We have a Problem tracking system, but it's not connected to TCS nor QA 
cycle in any way. 

We also have an automated TCS released by Borland with around 390 tests 
(please refer to the attached text for complete list of test groups and 
number of tests in each group). Tests are stored in FB database and test 
cases do not cover even major functionality areas, and mostly fall to the 
first test category. Tests mostly use ESQL for access to the database, 
instead of DSQL which is prevailing approach today. More to that, we 
cannot trust many tests, as it seems that the common practice was to 
write a test case and then catch the output (from tested engine!! in case 
of new features). This approach doesn't warrants the "correctness". This 
can be achieved only by human labour, i.e. someone have to get the input 
data, apply the tested algorythm on it and write down the expected 
results. This should be also verified by another person. It's obvious 
that declarative approach for input data and expected result would work 
better than procedural one. Well, it's a paranoid approach that assume 
that tested software cannot be trusted, but I can't imagine any other 
"less paranoid" approach to QA that would work for mission-critical 
software.  

TCS is hard to operate, and writing new tests require almost the same 
skills (or more in certain areas) for QA people as for core developers.  
The entry barrier for any apprentice QA developer is very high.


Some notes: 
For anything near the QA "certification", we'll absolutely must cover 
testing in area 1) in full and most important cases in 3) and 4). Other 
areas can follow later, but have to be considered when we'll plan our QA 
strategy and tools used.  

We'll need reference tests for SQL, stored procedure language and DSQL 
(most used) interface.  

All expected results of tests for correctness have to be "correct". 

We'll need more friendly TCS. Open source QMTest (www.codesourcery.com) 
is an option, but you can suggest other tools (including home-grown) as 
well.

Questions for you:
1) Do you think that current problem tracking system (and how we use it) 
is suitable for our needs ? 


2) If not, what changes do you recommend (technical or procedural) ?


3) What's your recommendations and requirements for TCS ? 


4) What test cases (groups) we should build and in what order/priority ?


5) Are you willing to write some test cases ? If you are, in what area ?


6) If yes, what practice do you suggest (will work best for you) ? This 
include requirements for TCS.




Your comments would be greatly appreciated.
Best regards
Pavel Cisar
http://www.ibphoenix.com
For all your upto date Firebird and
InterBase information

[Firebird-test] Firebird QA

[Pavel C. <pc...@us...>]

Hi all,

Thank you about your interest in Firebird QA efforts! I'm very sorry 
about my silence after I invited you to this discussion, but I was forced 
to do other things not related with Firebird project. But it seems that 
I'll have more than enough time now to continue with this topic in next 
weeks, and because the FB2 development is going onwards and we can hope 
for 1.5 release in next few months, this topic (and real progress with 
it) is more important than ever.   

First and foremost, I'd like inform you in detail about the current state 
of Firebird QA, and ideas and work we did in last months to improve it.

As you may know, quality is not an easy target, and there are many books 
and techniques to achieve it in software projects. Because a list of QA 
techniques that are currently used (or those we think that should be 
used) in Firebird project is quite long, I'm going to start a separate 
message thread for each technique to keep discussion clean and on topic.

But here you are a brief version of this list. 

a) White-box testing (WBT) by developers

1) Peer review.
2) Code audit
3) Development test cases.
4) Design by Contract.

Problems discovered by developers are rarely tracked in Bug Tracking 
System, and are fixed immediately.

b) Black-box testing (BBT) by "QA department"

The purpose of "QA department" is to verify correctness and limits of 
final product. That mean:

1) Correct input produce correct result.
2) Incorrect input produce correct error result.
3) Product reacts correctly to limit conditions and values (memory, 
disk space, file size, values at behaviour-switching boundary etc.)
4) Product reacts correctly to unexpected conditions (power failure, 
low-memory, I/O error)
5) Consistency. Test results from 1)-4) are consistent in time and 
with different order of execution. This also means concurrency tests.
6) Usability (performance, ergonomy, documentation)

If possible, automated test systems are used to do 1),2),3) and 5) 
tests. 4) and 6) usually depends completely on manual labour. All 
bugs and issues detected by QA department are tracked in some 
sort of Problem resolution system. This system is a crucial part of 
whole QA/development cycle, and any implemented QA procedure 
can't work successfully for long time without it. Due to its nature, I'll 
break this topic to next threads:

1) Automated QA - Test Control System issues
2) Problem tracking
3) Organization and procedural issues


If you have any comments or suggestions related to this lists, don't 
hesitate to share them in direct reply to this message.

Best regards
Pavel Cisar
http://www.ibphoenix.com
For all your upto date Firebird and
InterBase information

[Firebird-test] WBT: Peer Review

[Pavel C. <pc...@us...>]

Hi all,

Peer review is most often cited asset of Open Source development model, 
especially in relation to quality. So it's only natural that Peer review 
has it's place in Firebird Project.

Pro's and Con's:
This technique doesn't always work due to reviewer's time constraints, 
but helps especially to avoid the low-level design flaws.

Implementation:
Changes in CVS are reviewed as they appear (see firebird-checkins mailing 
list and newsgroup) by other developers. All problems are immediately 
reported and discussed in firebird-devel.

Current status:
We have that system already in place, because many changes made to the 
Firebird source tree are subject of review by at least one developer. But 
at least some (I hope that only some) changes are not checked at all. 
Actually, we haven't any shared knowledge, that peer review is really 
utilized in our project. Of course, some bugs were catched that way, but 
we don't know from others to what degree they do the review of changed 
code (if at all). There is not any developer designated to this task.

Questions for you:
1) Do you think that we should use peer review ? 


2) If yes, is the current state satisfactory for you ? 


3) If not, what changes do you recommend ?


Your comments would be greatly appreciated.
Best regards
Pavel Cisar
http://www.ibphoenix.com
For all your upto date Firebird and
InterBase information

[Firebird-test] TCS ported to SINIX-Z

[Erik K. <Eri...@ph...>]

Hi everyone,

I'd just let you know that I successfully ported TCS to SINIX-Z.
The changes have been applied to CVS. Does anyone announce this
on the FB website?


-- 
   Dipl.-Ing. Erik Kunze               Phone: +49 - 89 - 32 14 07 41
   PHILOSYS Software GmbH              Fax:   +49 - 89 - 32 14 07 12
   Edisonstr. 6                        Email: Eri...@ph...
   D-85716 Unterschleissheim           WWW:   www.philosys.de/~kunze

 PGP-Key: http://blackhole.pca.dfn.de:11371/pks/lookup?op=get&search=0xD5759581

Re: [Firebird-test] Re: Problems with DSQL_DOMAIN_08

[Frank Schlottmann-G. <sch...@t-...>]

Am Mittwoch, 26. Juni 2002 15:14 schrieb Erik Kunze:

> That's not the soulation, cause the test looks in WHERE_FILES for the
> test DB. I see two possible solutions:
>  o copy atlas.gbk from ./tests/sov3v4files to ./tests
>  o edit test and change WHERE_FILES into WHERE_EXAMPLES_40

From tcs_install.sh :
...
ln -s sov3v4files liv3v4files
...
cp -r TCS/test-files/* FB_test/tests
cp FB_test/tests/liv3v4files/atlas.gbk FB_test/tests/atlas.gbk
...

Frank

[Firebird-test] Re: Problems with DSQL_DOMAIN_08

[Erik K. <Eri...@ph...>]

John,

> > It expects the database to be in WHERE_FILES. WHERE_EXAMPLES_40 is set
> > to ./tests/sov3v4files. Should I modify the test or copy the file 
> > atlas.gbk
> > from ./tests/sov3v4files into ./tests directory?
> 
> Try updating WHERE_EXAMPLES_40 in your local environment to point to the 
> correct location.

That's not the soulation, cause the test looks in WHERE_FILES for the 
test DB. I see two possible solutions:
 o copy atlas.gbk from ./tests/sov3v4files to ./tests
 o edit test and change WHERE_FILES into WHERE_EXAMPLES_40

-- Erik

Re: [Firebird-test] Problems with DSQL_DOMAIN_08

[Frank Schlottmann-G. <sch...@t-...>]

Am Montag, 24. Juni 2002 12:03 schrieb Erik Kunze:
> Hi,
>
> I get the following output while running TCS test DSQL_DOMAIN_08 against
> FB1:

Could you try to get
http://firebird.sourceforge.net/download/TCS/TCS_linux.tar
and take a look at  tcs_install.sh,
the necessary symlinks for FB1 are all there (hopefully this script should 
even work for you :-)

Frank

Re: [Firebird-test] Problems with DSQL_DOMAIN_08

[John B. <bel...@cs...>]

Erik,

On Monday, June 24, 2002, at 03:03 AM, Erik Kunze wrote:

> Hi,
>
> I get the following output while running TCS test DSQL_DOMAIN_08 against
> FB1:
>
> [...]
> It expects the database to be in WHERE_FILES. WHERE_EXAMPLES_40 is set
> to ./tests/sov3v4files. Should I modify the test or copy the file 
> atlas.gbk
> from ./tests/sov3v4files into ./tests directory?

Try updating WHERE_EXAMPLES_40 in your local environment to point to the 
correct location.

-John