RE: [Springframework-developer] App Testing

[Alef A. \(JTeam\) <al...@jt...>]

>  * Tests should be repeatable and the results unambiguous.  
> This *may* involve 
> writing specific new test applications that focus on these 
> goals rather than 
> attempting to fit existing sample applications to the purpose 
> which may have 
> more of a user-educational benefit, or which are harder to 
> test in a regular 
> fashion.
Sounds reasonable to me. The goal is to test Spring in production
environments, not to test the sample applications.

>  * Test applications should be capable of automated 
> deployment and testing on 
> various J2EE server platforms.
Well, that sounds handy, otherwise people must be involved every test :)
 
> i. Given that no-one is likely to come forward with a large 
> array of high 
> powered hardware for our testing pleasure (but do feel free 
> to correct me), 
> the tests will need to be distributable.  We will probably be 
> able to find a 
> semi-permanent home for testing Tomcat/JBOSS, but this is the 
> only way we 
> will be able to test against many other server platforms - in 
> particular the 
> heavyweight and expensive commercial ones.  (SourceForge do 
> have a WebSphere 
> box in their compile farm, but I think it may be difficult to 
> run the sort of 
> tests we want to run there.)  
We're running a fully automated build here on three servers (well, you
can hardly call them servers, they're PIII-800s with 512-1024 ram). One
is the CVS server, one is the server building the projects, one of the
dedicated test server. Right now we're testing on a jboss/jetty combo
(tens of automated builds a day), but we've also been able to do
weblogic/oracle combo concurrently. It _is_ going to be a problem to
test WebSphere/9iAS on such machines however... If we need resources,
I'll probably be able to free them...

> ii. With distributable tests running ad-hoc on donated server 
> resources, it 
> may not be possible or desirable to operate tests that begin 
> with clean 
> platform installs.  In some cases, the platform install may 
> simply be too 
> problematic to script anyway, even if time and resource 
> permitted.  How 
> critical is this as a requirement?  Whilst certainly feasible on 
> Tomcat/JBOSS, I'd suggest that trashing and re-deploying the 
> test apps on an 
> existing server install is the best we might get for WebLogic 
> / WebSphere et 
> al.
I don't know how the other servers do it, as far as JBoss is concerned,
the server-configuration directoyr is enough. We could just put that in
the CVS ([jbosshome]/server/default for instance). We're having a
standard jboss-installation on the testserver and we're updating the
configuration directory each build (from the cvs). This works perfeclty.
As I said, I don't know how weblogic/websphere/9iAS are handling this.

> iii. Each application server will require it's own script(s), 
> perhaps even one 
> per test application per server.  However, once a test application is 
> deployed, the actual test should be designed to be handled in a 
> server-agnostic manner unless deliberately testing specific platform 
> differences.  Further investigation might show that something 
> like custom ANT 
> tasks would be of use somewhere here.
Agree...

> iv. Dependencies need careful consideration in how they may 
> affect test 
> results given a widespread test environment.  The database 
> problem has 
> already been remarked upon, local dependencies such as db 
> drivers, different 
> versions of libraries in the lib/ext (or equivalent) 
> directory of the server 
> installation and even possibly different JDK versions are 
> elements that will 
> have to be dealt with which wouldn't necessarily be issues in 
> a testing lab.  
> It may be possible to report some of this information within 
> the test results 
> where sensible values can be retreived.  Does anyone have 
> thoughts on this?
JUnit does this automatically for the VM performing the test, doesn't
it? As far as server configs are involved, are the tests going to move
around across server that much to make it desirable to automate this?