This particular WEP is very important to me. I think it's key to making
Webware a major league player in the application server market. I would
appreciate it if as many people as possible could scrutinize it and
critique it. You will ultimately benefit from your suggestions as this will
all be used to ensure the quality of Webware.
Author: Chuck Esterbrook
Webware, and especially WebKit, needs comprehensive, fully automated
regression testing. This is essential for stability, rapid development and
the 1.0 release of Webware.
This WEP outlines the details of what that would look like and how it would
Each Webware component should have a Tests/ directory containing Test*.py
files. Each one should be executed independently, producing no output
unless an error occurs. All errors should be reported via exceptions.
PyUnit (http://pyunit.sourceforge.net/) should be considered for this task.
Webware itself should have a test.py that kicks off the entire process:
> cd Webware
> python bin/test.py
In addition to basic unit tests, WebKit is sorely in need of live
regression testing on the app server in various configurations.
In conducting such a test, all URLs on the Examples and Admin pages should
be hit and checked for:
- HTTP status
- correct redirect, if expected
- correct content if expected
- correct cookies
Areas to cover:
- The Testing/ context of WebKit provides a starting point for test cases.
- Fields should be passed in.
- Bad urls:
- with no extension
- with a known extension (.py .psp)
- with an unknown extension (.skdfg)
This test program should accept the beginning URL such as:
The test program should be able to:
- be configured
- run on a machine other than the app server and web server
- log results and e-mail error messages
- record performance benchmarks
- run requests concurrently
should be considered for this task.
A "meta-test" program should provide for:
- Change each of the settings of the WebKit configuration files
- Run the regression test for each permutation of settings
- Relaunch the web server and/or app server as needed
- Switch adapters and set ups (One Shot, CGI, FastCGI, etc.)
A separate stress test should simply hit the server at increasingly
concurrent settings as fast as possible. ApacheBench is what I directly
have in mind for this. The stress test should be able to detect if things
go bad and report the problem.
In the test machine set up, the client should run on a separate machine
from the servers, since that is the only realistic situation for production
sites with heavy loads. e.g., web clients and web servers do not run on the
same machine except when developing.
The performance benchmarks should be collected in such a fashion that they
can be compared and graphed over nights and weeks. A sudden slow down in
performance is also a bug, but a more subtle one. gnuplot is a potential
tool for creating the performance graphs. Also, we might consider sending
e-mail alerts if a performance benchmark changes by a certain percentage.
Major op sys's that should be tested:
- UNIX family
Major web servers that should be tested:
On a nightly basis, a fresh copy of the CVS should be checked out and fully
tested with all of the above.
In the first versions of the regression tests, we will not cover
everything. On an on-going basis, we should review every method of every
class and every setting of every config file and ask, "Does the regression
test cover that?"
Also, as new features are added, they need to be accompanied by new
regression tests. If the developer doesn't have time to write every test he
can think of, he should at least write a sufficient number of initial tests
AND document ideas for further tests.
Chuck Esterbrook can dedicate hardware for the nightly tests on Linux & Apache.
Someone else will have to provide the Windows test machines.
Creating a test suite that covers all of the above is quite an undertaking.
As of 1/25 we have no takers or timelines. Chuck Esterbrook will dive in
eventually, but has no prediction when that will happen.
We should continue to flesh out this WEP in the mean time. Any and all
feedback is appreciated.