Re: [Springframework-developer] App Testing

[Rod J. <rod...@in...>]

Darren

>  * The main goal as I understand it is to provide an application testing
> sandbox - ie one that focuses on the usage of Spring and not on the Spring
> classes or components themselves.  In this way we hope to verify that
changes
> to Spring which are known not to break compilation or unit testing of
Spring
> itself, also can be shown not to break applications depending on it and/or
> maintain backward compatibility.

Yes

>  * While such tests may be more important as new releases approach, there
> should be no impediment (time and machine resources notwithstanding) to
> performing such tests nightly against CVS snapshots.

Yes

>  * Tests should be repeatable and the results unambiguous.  This *may*
involve
> writing specific new test applications that focus on these goals rather
than
> attempting to fit existing sample applications to the purpose which may
have
> more of a user-educational benefit, or which are harder to test in a
regular
> fashion.

Yes, but it should be possible to add more apps to the suite as necessary.

>  * Results should be made public in a consistent, user-friendly format,
> possibly via html and with mail notifications to the dev-list.

Yes. Simple success or failure, and information about any failure.

>
>  * Test applications should be capable of automated deployment and testing
on
> various J2EE server platforms.

Yes.


> i. Given that no-one is likely to come forward with a large array of high
> powered hardware for our testing pleasure (but do feel free to correct
me),
> the tests will need to be distributable.  We will probably be able to find
a
> semi-permanent home for testing Tomcat/JBOSS, but this is the only way we
> will be able to test against many other server platforms - in particular
the
> heavyweight and expensive commercial ones.  (SourceForge do have a
WebSphere
> box in their compile farm, but I think it may be difficult to run the sort
of
> tests we want to run there.)
>
> ii. With distributable tests running ad-hoc on donated server resources,
it
> may not be possible or desirable to operate tests that begin with clean
> platform installs.  In some cases, the platform install may simply be too
> problematic to script anyway, even if time and resource permitted.  How
> critical is this as a requirement?  Whilst certainly feasible on
> Tomcat/JBOSS, I'd suggest that trashing and re-deploying the test apps on
an
> existing server install is the best we might get for WebLogic / WebSphere
et
> al.

I agree. With JBoss/Tomcat I'd unzip it. Also Orion if we wanted to. I don't
know much about WebSphere configuration. WebLogic has nice clean separate
server config directories, so we should be able to handle it at that level.

> iii. Each application server will require it's own script(s), perhaps even
one
> per test application per server.  However, once a test application is
> deployed, the actual test should be designed to be handled in a
> server-agnostic manner unless deliberately testing specific platform
> differences.  Further investigation might show that something like custom
ANT
> tasks would be of use somewhere here.

Good point. We need to choose a test tool. RMI is handy for EJB-based tests.
Other stuff may need to be web tested via Latka or the like. Or we could
deploy some test cases into the server. I'm not a big fan of things like
Cactus in testing applications, but they may make sense for this kind of
thing. We could have some tests outside any container of course. Like
checking transactional behaviour via AOP against a database, and checking
the results.

> iv. Dependencies need careful consideration in how they may affect test
> results given a widespread test environment.  The database problem has
> already been remarked upon, local dependencies such as db drivers,
different
> versions of libraries in the lib/ext (or equivalent) directory of the
server
> installation and even possibly different JDK versions are elements that
will
> have to be dealt with which wouldn't necessarily be issues in a testing
lab.
> It may be possible to report some of this information within the test
results
> where sensible values can be retreived.  Does anyone have thoughts on
this?

We should be able to say which versions of a server we've tested on. E.g.
WebLogic 8.1 SP1, JBoss 3.x.x. We can't guarantee users testing on every
combination out there, but we should be able to say "that problem doesn't
occur with certain combinations we've tested."

>
> v. Unambiguous results may be a goal too far.  It may be that results will
> have to be verified individually with someone deciding whether a 'failure'
is
> really a failure that we need to be concerned with, or just a failure that
> was caused by 'that dodgy db driver again'.

SolarMetric have had massive issues with this in setting up their test
suites. There are a lot of buggy JDBC drivers out there when you start to
push them hard. However, I don't think we need to aim to test every database
out there. I'd be happy just to use an embedded database to start off.
Greatly simplifies the config process also.

I think a first goal might be Tomcat + HSQL. Easy to do the "just unzip"
approach and meets the basic goal of verifying that we didn't break
everyone's web app in changes today.

The goal of saying "Spring works end to end on app server X" is a separate,
but also important, goal, I think.

Regards,
Rod