[Springframework-developer] Testing

[Rod J. <rod...@in...>]

Hi Guys,

I've been thinking about how to improve our testing strategy further.

Currently we have excellent--and continually improving--unit test coverage.
This is essential and, I believe, the most important form of testing for a
framework such as Spring.

However it would also be good to have automated integration tests that help
to verify that code changes won't break user applications. This is
especially important before releases, but would be really useful possibly as
part of a nightly build process. (I don't see any point running unit tests
in a nightly build: developers shouldn't commit without doing that.) I think
it's important to move beyond ad hoc testing, not just for unit testing.

I recently had lunch with Patrick Linskey of SolarMetric (Kodo JDO). He was
telling me about their testing, which is impressive. They have multiple
machines devoted to continually testing Kodo on different application
servers and against different databases. They start from a clean install of
the app server, then install Kodo and their test applications.

Perhaps we could apply a similar approach as we move towards 1.0 final.

The basic process would probably involve an additional Ant script, and look
something like this:

- be able to do a checkout from a tag or the head
- compile and build deployment units
- take a clean application server (perhaps tarred) and deploy the spring
Jars (and dependencies) from CVS along with the target application from CVS
(one of our samples, at least to start with). We'd also need a database set
up similarly--probably HSQL or such an embedded database, including create
- wait until deployment is complete, unless the target app server can tell
us
- run tests against the app. This might involve HTTP requests with something
like HTTPUnit. Or maybe there are better tools. If there are any EJB
endpoints in any app we could do RMI against them.
- clean up so that no temporary files would break the next run

There are a number of challenges and a lot of work to get to this point, but
I think it would be worth it. I would love to get Spring a reputation for
stability and lack of bugs. Tasks would include:

- Writing build script
- Deciding which app servers to target? I'd suggest Tomcat and JBoss or
WebLogic to start off with. Also, where do we keep the app server tarballs?
We can't check JBoss or WebLogic into CVS.
- Deciding where do these tests run? How often do they run? Who looks at the
results? Maybe email would be the best way to go.
- Writing test scripts for deployed apps, and deciding on the best tool.

Ideally the scripts would allow adding new app servers or sample
applications + test scripts, within a consistent framework.

I'd like some views on this. Does anyone have experience of doing something
similar?

Also, a volunteer to take ownersip would be great, assuming we agree it's a
good idea! This might be a good opportunity for someone who doesn't
presently have CVS access but wants to be a Spring developer, as it wouldn't
involve changing existing code. (Not that I want to discourage any present
developer.)

Regards,
Rod