Status

During the first year, FITTEST has made considerable progress and reached all of the planned objectives! The following achievements were made:

• We have developed a stable prototype of the FITTEST Integrated Testing Environment (ITE) for Continous Testing delivered as an Eclipse plugin. Various components for logging and oracle-inference (WP6), model inference (WP4) and generation of combinatorial trees (WP5) have been integrated (see below). The prototype enables to perform a whole FITTEST cycle (see Global Picture) for Habbo-like Flash applications and PHP-applications From the logging of the end-users' interactions with a Flash application, the automated generation of a model, the automated generation of abstract test cases and the replaying of generated concrete test cases. This prototype of the FITTEST ITE has been installed and tested at industrial parters Sulake, Softeam, IBM and B&M.

• We have defined a general methodological framework for evaluating software testing techniques and tools and instantiated it once at each case study partner (Sulake, Softeam and IBM) to define and execute 3 case studies that evaluate the effectiveness, efficiency and subjective satisfaction of some selected FITTEST techniques.

• Investigated the use and necessity of benchmarking for evaluating testing techniques, defined a template for defining benchmarks and instantiated this template with four benchmarks: one for continuous testing, another for combinatorial testing, one for concurrency testing and finally one for mutation testing.

• Created a co-evolution framework evolving higher order mutants and test cases. In each iteration the mutation tool developed by UCL evolves a unique set of higher order mutants. At the same time, our test data generation tool tries to generate test data to weakly kill the mutants. The outcome of this co-evolution are a set of higher-order mutants that are hard to kill and test data that is good at killing many mutants (and thus good at finding bugs)

• Created a road-map paper for search-based software engineering in/for the cloud (submitted for publication in Journal of Systems and Software). The paper discusses different challenges for cloud clients (consumers of cloud infrastructure) and cloud providers. It also lists five problems that are amenable for search-based software engineering and proposes solution representation and fitness functions for these problems.

• Created a fully automatic GUI testing framework which can be used to test the robustness of large and complex applications at the GUI level (w.r.t. crashes).

• Developed and implemented Numeric Dependency Rules, Dynamic Dependency Rules, Test Sequence Generation, Optimization of Prioritized Test Suites, General Test Execution, Evaluation for Test-Results and new user interfaces in the CTE XL Professional and have published the results.

• Developed a new ConcurrencyTesting capability of generating intense artificial load for concurrency testing in large-scale distributed application as typically found in FI. This new capability builds upon the user-defined target methods feature developed in the first reporting period.

• Enhanced the ConcurrencyTesting technology with debugging capabilities of concurrent bugs in FI applications with Orange Box, record and replay, and new lock reporting capabilities.

• Developed technology for re-creating segments of valuable realistic test cases form previous user interaction, in order to repair test cases and test out new FI functionality.

• Developed a regression test case prioritization technology which was tested against a FITTEST case study and initial results show good prioritization.