DesignRequirements

Design Requirements

PEST's design is the culmination of a lot of experience with distributed system testing. As always, mistakes are the best way to learn, so PEST corrects many of the mistakes inherent in other test harnesses by getting the requirements right.

1. Leverage existing scripts.
2. Don't prohibit tests being written in any language.
3. Support Standard Testbed Descriptor to allow test portability.
4. Remote process invocation.
5. Minimize output buffering.
6. Support threading/backgrounding.
7. Allow continuous manual monitoring of individual test processes.
8. Clear separation between the harness and test scripts with a standard interface contract. Execute test scripts outside of the harness for debugging and reproduction.
9. Support standard execution synopsis.
10. Clear separation of log metadata and data and harness debug logs.
11. Easy debugging of individual processes.
12. View exact command issued.
13. Easy means to map scripts to test case database IDs.
14. Easy to create a test case batch, mapped to a test case database 'run' or 'suite'.
15. Support positive and negative test cases.
16. Support system (parallel) and functional (serial) testing.
17. Test execution should not require development skills.
18. All config files must be copied along with the logs for reproducability.
19. Logs should tell the user what's going on with the systems under test and not be obfuscated by framework control flow progress.
20. The harness should optionally send the user an email with a summary of the test results and link to the logs.
21. Harness log and harness output must summarize process activities and report totals.
22. Harness log and harness output must report activities prior to execution.
23. Harness log and harness output must report run metadata, including log paths at the beginning and end of the output.
24. Minimal external module dependencies.
25. Architecture must allow for local or network installation of the harness, the test framework(s), and the logs.