[Firebird-test] Firebird QA

[Pavel C. <pc...@us...>]

Hi all,

Thank you about your interest in Firebird QA efforts! I'm very sorry 
about my silence after I invited you to this discussion, but I was forced 
to do other things not related with Firebird project. But it seems that 
I'll have more than enough time now to continue with this topic in next 
weeks, and because the FB2 development is going onwards and we can hope 
for 1.5 release in next few months, this topic (and real progress with 
it) is more important than ever.   

First and foremost, I'd like inform you in detail about the current state 
of Firebird QA, and ideas and work we did in last months to improve it.

As you may know, quality is not an easy target, and there are many books 
and techniques to achieve it in software projects. Because a list of QA 
techniques that are currently used (or those we think that should be 
used) in Firebird project is quite long, I'm going to start a separate 
message thread for each technique to keep discussion clean and on topic.

But here you are a brief version of this list. 

a) White-box testing (WBT) by developers

1) Peer review.
2) Code audit
3) Development test cases.
4) Design by Contract.

Problems discovered by developers are rarely tracked in Bug Tracking 
System, and are fixed immediately.

b) Black-box testing (BBT) by "QA department"

The purpose of "QA department" is to verify correctness and limits of 
final product. That mean:

1) Correct input produce correct result.
2) Incorrect input produce correct error result.
3) Product reacts correctly to limit conditions and values (memory, 
disk space, file size, values at behaviour-switching boundary etc.)
4) Product reacts correctly to unexpected conditions (power failure, 
low-memory, I/O error)
5) Consistency. Test results from 1)-4) are consistent in time and 
with different order of execution. This also means concurrency tests.
6) Usability (performance, ergonomy, documentation)

If possible, automated test systems are used to do 1),2),3) and 5) 
tests. 4) and 6) usually depends completely on manual labour. All 
bugs and issues detected by QA department are tracked in some 
sort of Problem resolution system. This system is a crucial part of 
whole QA/development cycle, and any implemented QA procedure 
can't work successfully for long time without it. Due to its nature, I'll 
break this topic to next threads:

1) Automated QA - Test Control System issues
2) Problem tracking
3) Organization and procedural issues


If you have any comments or suggestions related to this lists, don't 
hesitate to share them in direct reply to this message.

Best regards
Pavel Cisar
http://www.ibphoenix.com
For all your upto date Firebird and
InterBase information