1. Introduction
2. Black Box GUI Automation Overview
3. Object Oriented Testing
4. Image Based Testing
5. Implementation Factors
6. Summary
This document compares object oriented and image based approaches of black box GUI automated software testing. It is intended to respond frequently asked questions about these two technologies and to provide basic information for those who are looking for automated testing solutions.
Hardware devices operated by humans nowadays have a lot in common, such as:
Figure 1-1: GUI enabled hardware devices
Most modern GUI applications for these devices are implemented on top of a programming platform, such as Java or .NET. Platforms allow developers to build complex GUIs from the platform's library of graphical primitives such as buttons, drop downs, panels and windows. When the application is being run, the platform (or code injected to the application by the platform) paints the application GUI image onto the display device, usually through the screen image buffer operated by the OS. HTML documents generated by web applications can be also considered GUIs save that their image gets painted by a browser rather than the underlying platform.
Software applications typically do not approach the keyboard and pointer devices directly. They rather act as consumers of keyboard and pointer events delivered by the underlying OS.
Figure 1-2: Application architecture
Black box GUI automated testing tools are software applications which behave in a slightly different way. They generate keyboard and pointer events or invoke GUI component actions in order to navigate through GUI of the tested application. They also allow to analyze the GUI to verify whether the application under test (AUT) behaves as expected.
Automated testing tools can be roughly classified by the way they approach the GUI to:
Object oriented testing benefits from tight integration with the programming platform. Tools based on this principle understand the platform's technology and they are able to identify individual GUI objects (components), read their properties and interact with them. They can handle test steps such as "verify that there are two buttons labelled 'OK' and 'Cancel'" or "find out whether the 'OK' button is enabled or disabled".
Object oriented testing tools typically invoke actions on the GUI components through the platform APIs. For example, to simulate a mouse click on a Java component JButton one may call the button's doClick() method. Some tools in this category are also able to generate the keyboard and pointer events on the OS level through posting to the system event queues. Verification of test results is usually done through checking of existence or state of a particular GUI component, such as "Check that there's a text field labelled 'Result' and contains the text of '1'".
Figure 2-1: Object oriented testing approach
Object oriented testing approach prevails among automated testing tools today thanks to:
There are however also many disadvantages, such as
Image based testing tools typically automate on the level of operating system, eventually on top of another layer allowing to access all the necessary devices (such as virtualization or remote desktop software). Automation is driven through keyboard and pointer events injected into the system event queues. GUI of the tested application is accessed through the system display buffer on the pixel level.
Verification is usually performed through image comparison methods, such as image search or object recognition.
Figure 3-1: Image based testing approach
Advantages of image based testing are quite obvious:
This testing concept also suffers from some minor disadvantages:
Choosing the right automated testing tool for your project is crucial. Before you make a decision, figure out what your test environment is like and what your expectations, resources and future development plans are. When deciding between the object oriented and image based technologies, consider the following factors:
From the capability point of view, object oriented testing focuses vertically on a particular programming technology and it performs well in single technology environments. Some test steps out of the particular technology however can't be usually automated with the same tool. Automation development costs are usually higher while maintenance costs are lower thanks to good robustness.
Image based testing is more generic and universal. It performs well in environments with mixed technologies and in certain cases where object oriented tools fail or lack support of a certain feature or technology. Automation development costs are typically lower because of simplicity, but long term maintenance costs may be higher due to environment vulnerabilities.