GlobalPic

As the FITTEST global picture shows, the FITTEST ITE for Continuous Testing integrates testing tools developed and/or improved as result of the tasks performed through the work packages 3 to 9 of the FITTEST EU project, including but not limited to:

• UCL WATT for Coverage testing
• IBM ConTest for Concurrency testing
• IBM Focus for Testware generation and Coverage testing
• B&M CTE for Combinatorial testing
• FBK TSS for Regression testing
• FITTEST Continuous Testing Tool
• GUITEST tool for GUI crash testing

In the global picture, the FITTEST Continuous Testing approach is outlined in more details. It roughly consists of four main stages in a loop (or cycle) as depicted in the picture above. Though the starting point in the cycle depends on the concrete testing scenario and the concrete quality criteria pursued, the typical full-cycle starts instrumenting the System Under Test (SUT). The instrumentation is usually needed for monitoring SUT execution and guiding the testing tasks.

The yellow boxes in the figure represent the evolutionary approach of FITTEST. That is, the FITTEST techniques are optimized when possible to deal with the nature of FI applications, using evolutionary search based software engineering techniques.

The application or System Under Test (SUT) is encapsulated into a Test Case Execution (TCE) environment that enables features of tracing and controlled-execution of the application. The environment consists of instrumentation facilities for the underlying technology and capture-replay utilities that help monitor the test cases generated and executed over the FITTEST cycle.

Three main modules are identified for the TCE environment. The logger feature keeps track of the events (system events, user events, exceptions risen, etc.) during executions of the SUT. The coverage module collects information necessary to measure the degree of testing achieved. The last module runs concurrency tests while the SUT is executed. Three main sources of information come from the TCE environment: logs, coverage and failures/bugs.

The information gathered from the SUT executions is analysed by low-level techniques. For example, from the logged information, FITTEST will do: log diagnosis, model inference and oracle learning. The last one may provide the facility of inferring oracles that predict which SUT executions may be reasonable and may be prone to errors detection. The other two (log diagnosis and model inference) produce artifacts for the next stage in the cycle: testware generation.

The high-level testing techniques (i.e. model-based testing and combinatorial testing) use the artifacts produced to automatically generate abstract test cases. For the combinatorial part, the end-user may interact to adapt the selection of test cases to specific criteria. Mutation testing is another high-level technique that can be used to generate test data using evolutionary test case generation techniques.

The last stage is responsible for compilation of the generated abstract test cases into concrete test cases that the SUT can execute in the context of the TCE. The compilation of the test cases can use oracles learned or other test data generated from evolutionary computations based on the logs, coverage and/or failure information.