[Springframework-developer] App Testing

[Darren D. <da...@da...>]

On Saturday 11 October 2003 18:29, Rod Johnson wrote:

> However it would also be good to have automated integration tests that help
> to verify that code changes won't break user applications. This is
> especially important before releases, but would be really useful possibly
> as part of a nightly build process. (I don't see any point running unit
> tests in a nightly build: developers shouldn't commit without doing that.)
> I think it's important to move beyond ad hoc testing, not just for unit
> testing.

--remainder snipped-- 


as promised a few days ago, I spent a little while thinking about this and 
there are some challenges involved to meet even the primary goals.  I'll 
outline some of my thoughts here, but I'd like to get some feedback from 
people on the list as to what is deemed really important and therefore 
worthwhile spending the time on before expanding much further.

 * The main goal as I understand it is to provide an application testing 
sandbox - ie one that focuses on the usage of Spring and not on the Spring 
classes or components themselves.  In this way we hope to verify that changes 
to Spring which are known not to break compilation or unit testing of Spring 
itself, also can be shown not to break applications depending on it and/or 
maintain backward compatibility.

 * While such tests may be more important as new releases approach, there 
should be no impediment (time and machine resources notwithstanding) to 
performing such tests nightly against CVS snapshots.

 * Tests should be repeatable and the results unambiguous.  This *may* involve 
writing specific new test applications that focus on these goals rather than 
attempting to fit existing sample applications to the purpose which may have 
more of a user-educational benefit, or which are harder to test in a regular 
fashion.

 * Results should be made public in a consistent, user-friendly format, 
possibly via html and with mail notifications to the dev-list.

 * Test applications should be capable of automated deployment and testing on 
various J2EE server platforms.


These are the primary aims as I see them presently.  If anyone has views of 
any kind on the primary aims, I'd be very keen to hear them via the list.  
Following on from these are some other thoughts that are influenced by 
implementation constraints, some of which are outright questions rather than 
suggestions for moving forward.

i. Given that no-one is likely to come forward with a large array of high 
powered hardware for our testing pleasure (but do feel free to correct me), 
the tests will need to be distributable.  We will probably be able to find a 
semi-permanent home for testing Tomcat/JBOSS, but this is the only way we 
will be able to test against many other server platforms - in particular the 
heavyweight and expensive commercial ones.  (SourceForge do have a WebSphere 
box in their compile farm, but I think it may be difficult to run the sort of 
tests we want to run there.)  

ii. With distributable tests running ad-hoc on donated server resources, it 
may not be possible or desirable to operate tests that begin with clean 
platform installs.  In some cases, the platform install may simply be too 
problematic to script anyway, even if time and resource permitted.  How 
critical is this as a requirement?  Whilst certainly feasible on 
Tomcat/JBOSS, I'd suggest that trashing and re-deploying the test apps on an 
existing server install is the best we might get for WebLogic / WebSphere et 
al.

iii. Each application server will require it's own script(s), perhaps even one 
per test application per server.  However, once a test application is 
deployed, the actual test should be designed to be handled in a 
server-agnostic manner unless deliberately testing specific platform 
differences.  Further investigation might show that something like custom ANT 
tasks would be of use somewhere here.

iv. Dependencies need careful consideration in how they may affect test 
results given a widespread test environment.  The database problem has 
already been remarked upon, local dependencies such as db drivers, different 
versions of libraries in the lib/ext (or equivalent) directory of the server 
installation and even possibly different JDK versions are elements that will 
have to be dealt with which wouldn't necessarily be issues in a testing lab.  
It may be possible to report some of this information within the test results 
where sensible values can be retreived.  Does anyone have thoughts on this?

v. Unambiguous results may be a goal too far.  It may be that results will 
have to be verified individually with someone deciding whether a 'failure' is 
really a failure that we need to be concerned with, or just a failure that 
was caused by 'that dodgy db driver again'.



I'll keep my further thoughts to myself for now, I'd be keen to see what the 
feeling is about all of this.

-- 

Darren Davison
Public Key: http://www.davison.uk.net/key.jsp