Once upon a time (about 10 years ago) developer Geoff Bache was working on an optimization system that involved processing enormous mathematical problems that could not be solved exactly. That meant he needed better and better heuristics to get a better solution as he went along. “It felt unnatural to write functional tests for the program using techniques based on unit testing,” he says, “since it was impossible to say in advance what the ‘right’ answer was and write an assertion for it.” The answer to that conundrum was TextTest, a functional testing tool with a twist.
TextTest is fundamentally different from most functional testing tools in that it runs the system under test from the command line and compares textual output from it with a previously checked version of that output. This leads to a more dynamic approach to testing, Bache says. “Essentially it exists to manage changes in the behavior of a system rather than trying to assert ‘timeless truths’ about it.” Bache says most test automation tools use the unit test paradigm of writing some code against an API and asserting that what you get back is equal to some hardcoded value. Functional test tools often add abstraction layers on top of this, but at heart they are basically doing the same thing.
You can use TextTest to test a program written in any language that terminates of its own accord and produces some text output that can be used as a measure of its behavior. It’s easiest to use with a batch program or script, but TextTest can also be used on GUIs in conjunction with a simulation tool, such as PyUseCase for Python GUIs. If you need to test a program that uses a database, you can write a small script to generate a textual description of the changes to the database during the test.
TextTest is particularly useful for testing a module with external dependencies. When creating a test you can automatically capture interactions with other modules (for instance system calls, synchronous messages over sockets, or ordinary Python method calls to other modules) and generate mock objects for them. When you run the test later, TextTest seamlessly replaces the external module with the mock version, which just replays the interaction it captured earlier. This saves you the work of hand-coding a mock, guarantees that the mock is behaving in a correct way, and makes it possible to test code in isolation.
“I find this useful for testing TextTest’s integration with Sun Grid Engine, version control systems, bug reporting systems, etc.,” Bache says. “It means I can quickly create tests that pass on systems where these tools are not even installed.”
Bache cautions developers “that it’s a different approach to testing than xUnit and that ‘red’ means the behavior has changed, not necessarily that it has failed. Remember that your tests are as readable and maintainable as the logs and other text files your system produces, and be prepared to invest in improving them for testability.”
TextTest is written in Python, using PyGTK for the user interface part. “I’ve always found Python to be a highly productive and expressive language, and the ability to ‘monkey-patch’ (modify code without touching the source itself) is very useful when testing. As for PyGTK, this came naturally as our main development platform is Linux.”
Bache chose to make the tool open source because “for a small investment in time and energy to make the external releases we get good feedback and contributions from the community. It has mostly been written on company time but my employer isn’t in the business of selling test tools, so there were never any serious plans to try and make money from selling it.”
Future plans for TextTest involve improving it as a test management tool to wrap around other popular test runners, and to improve its interactions with tools such as bug trackers, version control systems, and logging frameworks. Bache would appreciate help with the project “especially in areas where we have little internal incentive to expend effort. For example, it works on Windows, but all the ‘corporate’ users are on Linux, which affects my focus. And being able to run many tests in parallel is a key feature, but it’s currently dependent on Sun Grid Engine; it would be great to expand this to be able to, for example, make use of the cloud.” If you’d like to jump in and help, send a message to the SourceForge mailing list.