springframework-developer

[Springframework-developer] Testing

[Rod J. <rod...@in...>]

Hi Guys,

I've been thinking about how to improve our testing strategy further.

Currently we have excellent--and continually improving--unit test coverage.
This is essential and, I believe, the most important form of testing for a
framework such as Spring.

However it would also be good to have automated integration tests that help
to verify that code changes won't break user applications. This is
especially important before releases, but would be really useful possibly as
part of a nightly build process. (I don't see any point running unit tests
in a nightly build: developers shouldn't commit without doing that.) I think
it's important to move beyond ad hoc testing, not just for unit testing.

I recently had lunch with Patrick Linskey of SolarMetric (Kodo JDO). He was
telling me about their testing, which is impressive. They have multiple
machines devoted to continually testing Kodo on different application
servers and against different databases. They start from a clean install of
the app server, then install Kodo and their test applications.

Perhaps we could apply a similar approach as we move towards 1.0 final.

The basic process would probably involve an additional Ant script, and look
something like this:

- be able to do a checkout from a tag or the head
- compile and build deployment units
- take a clean application server (perhaps tarred) and deploy the spring
Jars (and dependencies) from CVS along with the target application from CVS
(one of our samples, at least to start with). We'd also need a database set
up similarly--probably HSQL or such an embedded database, including create
- wait until deployment is complete, unless the target app server can tell
us
- run tests against the app. This might involve HTTP requests with something
like HTTPUnit. Or maybe there are better tools. If there are any EJB
endpoints in any app we could do RMI against them.
- clean up so that no temporary files would break the next run

There are a number of challenges and a lot of work to get to this point, but
I think it would be worth it. I would love to get Spring a reputation for
stability and lack of bugs. Tasks would include:

- Writing build script
- Deciding which app servers to target? I'd suggest Tomcat and JBoss or
WebLogic to start off with. Also, where do we keep the app server tarballs?
We can't check JBoss or WebLogic into CVS.
- Deciding where do these tests run? How often do they run? Who looks at the
results? Maybe email would be the best way to go.
- Writing test scripts for deployed apps, and deciding on the best tool.

Ideally the scripts would allow adding new app servers or sample
applications + test scripts, within a consistent framework.

I'd like some views on this. Does anyone have experience of doing something
similar?

Also, a volunteer to take ownersip would be great, assuming we agree it's a
good idea! This might be a good opportunity for someone who doesn't
presently have CVS access but wants to be a Spring developer, as it wouldn't
involve changing existing code. (Not that I want to discourage any present
developer.)

Regards,
Rod

Re: [Springframework-developer] Testing

[<tri...@tr...>]

Rod & All,

I think automated integration testing is a great idea.  I would suggest that we
start with a JBoss/HSQL combo - easy install and the db is integrated in the app
server.  Later on we could add a WebLogic/Oracle combo.  Here we might not want
to install the server software for each test, but we could create a WebLogic
domain and an Oracle instance/database using scripts for each test.

Speaking of tests - I have not been able to run the entire test suite lately. 
It keeps failing on the org.springframework.metadata.bcel.AttributeWriterTests.
 I can run this test by itself OK under Eclipse, but not using the 'tests'
target in the build script.  Has anybody else seen this?  Here is the error:
Testsuite: org.springframework.metadata.bcel.AttributeWriterTests
Tests run: 1, Failures: 1, Errors: 0, Time elapsed: 0.771 sec

Testcase: testAttributes took 0.771 sec
	FAILED
Expected one custom class attribute expected:<1> but was:<0>
junit.framework.AssertionFailedError: Expected one custom class attribute
expected:<1> but was:<0>
	at
org.springframework.metadata.bcel.AttributeWriterTests.testAttributes(AttributeWriterTests.java:74)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

Testcase: testAttributes

Not sure what is going on here.

Thomas

> Hi Guys,
> 
> I've been thinking about how to improve our testing strategy further.
> 
> Currently we have excellent--and continually improving--unit test coverage.
> This is essential and, I believe, the most important form of testing for a
> framework such as Spring.
> 
> However it would also be good to have automated integration tests that help
> to verify that code changes won't break user applications. This is
> especially important before releases, but would be really useful possibly as
> part of a nightly build process. (I don't see any point running unit tests
> in a nightly build: developers shouldn't commit without doing that.) I think
> it's important to move beyond ad hoc testing, not just for unit testing.
> 
> I recently had lunch with Patrick Linskey of SolarMetric (Kodo JDO). He was
> telling me about their testing, which is impressive. They have multiple
> machines devoted to continually testing Kodo on different application
> servers and against different databases. They start from a clean install of
> the app server, then install Kodo and their test applications.
> 
> Perhaps we could apply a similar approach as we move towards 1.0 final.
> 
> The basic process would probably involve an additional Ant script, and look
> something like this:
> 
> - be able to do a checkout from a tag or the head
> - compile and build deployment units
> - take a clean application server (perhaps tarred) and deploy the spring
> Jars (and dependencies) from CVS along with the target application from CVS
> (one of our samples, at least to start with). We'd also need a database set
> up similarly--probably HSQL or such an embedded database, including create
> - wait until deployment is complete, unless the target app server can tell
> us
> - run tests against the app. This might involve HTTP requests with something
> like HTTPUnit. Or maybe there are better tools. If there are any EJB
> endpoints in any app we could do RMI against them.
> - clean up so that no temporary files would break the next run
> 
> There are a number of challenges and a lot of work to get to this point, but
> I think it would be worth it. I would love to get Spring a reputation for
> stability and lack of bugs. Tasks would include:
> 
> - Writing build script
> - Deciding which app servers to target? I'd suggest Tomcat and JBoss or
> WebLogic to start off with. Also, where do we keep the app server tarballs?
> We can't check JBoss or WebLogic into CVS.
> - Deciding where do these tests run? How often do they run? Who looks at the
> results? Maybe email would be the best way to go.
> - Writing test scripts for deployed apps, and deciding on the best tool.
> 
> Ideally the scripts would allow adding new app servers or sample
> applications + test scripts, within a consistent framework.
> 
> I'd like some views on this. Does anyone have experience of doing something
> similar?
> 
> Also, a volunteer to take ownersip would be great, assuming we agree it's a
> good idea! This might be a good opportunity for someone who doesn't
> presently have CVS access but wants to be a Spring developer, as it wouldn't
> involve changing existing code. (Not that I want to discourage any present
> developer.)
> 
> Regards,
> Rod
> 
> 
> 
> 
> -------------------------------------------------------
> This SF.net email is sponsored by: SF.net Giveback Program.
> SourceForge.net hosts over 70,000 Open Source Projects.
> See the people who have HELPED US provide better services:
> Click here: http://sourceforge.net/supporters.php
> _______________________________________________
> Springframework-developer mailing list
> Spr...@li...
> https://lists.sourceforge.net/lists/listinfo/springframework-developer
> 

RE: [Springframework-developer] Testing

[Alef A. \(JTeam\) <al...@jt...>]

About the server install, if you're starting with Jboss, the only thing
you need basically is the server configuration (of course besides the
normal binaries).

We have a mechanism like this in place here (for our automated build
server) and it works perfectly. Currently we're running one project
against a Jboss322RC4/MySQL combo and one against a Jboss/Oracle combo,
both having automated install of the Jboss configuration (the
[jboss]/server/[config]-dir which resides in the CVS).

How do we plan on doing this? Of course a Jboss/HSQL combo is not that
difficult, but if we also want to do WebLogic Oracle, automated installs
and stuff won't be that easy I think. Do we let it run at one of our
places or something?

Alef

-----Oorspronkelijk bericht-----
Van: spr...@li...
[mailto:spr...@li...] Namens
tri...@tr...
Verzonden: Saturday, October 11, 2003 11:21 PM
Aan: Rod Johnson
CC: spr...@li...
Onderwerp: Re: [Springframework-developer] Testing


Rod & All,

I think automated integration testing is a great idea.  I would suggest
that we start with a JBoss/HSQL combo - easy install and the db is
integrated in the app server.  Later on we could add a WebLogic/Oracle
combo.  Here we might not want to install the server software for each
test, but we could create a WebLogic domain and an Oracle
instance/database using scripts for each test.

Speaking of tests - I have not been able to run the entire test suite
lately. 
It keeps failing on the
org.springframework.metadata.bcel.AttributeWriterTests.
 I can run this test by itself OK under Eclipse, but not using the
'tests' target in the build script.  Has anybody else seen this?  Here
is the error:
Testsuite: org.springframework.metadata.bcel.AttributeWriterTests
Tests run: 1, Failures: 1, Errors: 0, Time elapsed: 0.771 sec

Testcase: testAttributes took 0.771 sec
	FAILED
Expected one custom class attribute expected:<1> but was:<0>
junit.framework.AssertionFailedError: Expected one custom class
attribute expected:<1> but was:<0>
	at
org.springframework.metadata.bcel.AttributeWriterTests.testAttributes(At
tributeWriterTests.java:74)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
a:39)
	at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)

Testcase: testAttributes

Not sure what is going on here.

Thomas

> Hi Guys,
> 
> I've been thinking about how to improve our testing strategy further.
> 
> Currently we have excellent--and continually improving--unit test 
> coverage. This is essential and, I believe, the most important form of

> testing for a framework such as Spring.
> 
> However it would also be good to have automated integration tests that

> help to verify that code changes won't break user applications. This 
> is especially important before releases, but would be really useful 
> possibly as part of a nightly build process. (I don't see any point 
> running unit tests in a nightly build: developers shouldn't commit 
> without doing that.) I think it's important to move beyond ad hoc 
> testing, not just for unit testing.
> 
> I recently had lunch with Patrick Linskey of SolarMetric (Kodo JDO). 
> He was telling me about their testing, which is impressive. They have 
> multiple machines devoted to continually testing Kodo on different 
> application servers and against different databases. They start from a

> clean install of the app server, then install Kodo and their test 
> applications.
> 
> Perhaps we could apply a similar approach as we move towards 1.0 
> final.
> 
> The basic process would probably involve an additional Ant script, and

> look something like this:
> 
> - be able to do a checkout from a tag or the head
> - compile and build deployment units
> - take a clean application server (perhaps tarred) and deploy the 
> spring Jars (and dependencies) from CVS along with the target 
> application from CVS (one of our samples, at least to start with). 
> We'd also need a database set up similarly--probably HSQL or such an 
> embedded database, including create
> - wait until deployment is complete, unless the target app server can
tell
> us
> - run tests against the app. This might involve HTTP requests with
something
> like HTTPUnit. Or maybe there are better tools. If there are any EJB
> endpoints in any app we could do RMI against them.
> - clean up so that no temporary files would break the next run
> 
> There are a number of challenges and a lot of work to get to this 
> point, but I think it would be worth it. I would love to get Spring a 
> reputation for stability and lack of bugs. Tasks would include:
> 
> - Writing build script
> - Deciding which app servers to target? I'd suggest Tomcat and JBoss 
> or WebLogic to start off with. Also, where do we keep the app server 
> tarballs? We can't check JBoss or WebLogic into CVS.
> - Deciding where do these tests run? How often do they run? Who looks 
> at the results? Maybe email would be the best way to go.
> - Writing test scripts for deployed apps, and deciding on the best 
> tool.
> 
> Ideally the scripts would allow adding new app servers or sample 
> applications + test scripts, within a consistent framework.
> 
> I'd like some views on this. Does anyone have experience of doing 
> something similar?
> 
> Also, a volunteer to take ownersip would be great, assuming we agree 
> it's a good idea! This might be a good opportunity for someone who 
> doesn't presently have CVS access but wants to be a Spring developer, 
> as it wouldn't involve changing existing code. (Not that I want to 
> discourage any present
> developer.)
> 
> Regards,
> Rod
> 
> 
> 
> 
> -------------------------------------------------------
> This SF.net email is sponsored by: SF.net Giveback Program. 
> SourceForge.net hosts over 70,000 Open Source Projects. See the people

> who have HELPED US provide better services: Click here: 
> http://sourceforge.net/supporters.php
> _______________________________________________
> Springframework-developer mailing list 
> Spr...@li...
> https://lists.sourceforge.net/lists/listinfo/springframework-developer
> 





-------------------------------------------------------
This SF.net email is sponsored by: SF.net Giveback Program.
SourceForge.net hosts over 70,000 Open Source Projects. See the people
who have HELPED US provide better services: Click here:
http://sourceforge.net/supporters.php
_______________________________________________
Springframework-developer mailing list
Spr...@li...
https://lists.sourceforge.net/lists/listinfo/springframework-developer

Re: [Springframework-developer] Testing

[Rod J. <rod...@in...>]

> We have a mechanism like this in place here (for our automated build
> server) and it works perfectly. Currently we're running one project
> against a Jboss322RC4/MySQL combo and one against a Jboss/Oracle combo,
> both having automated install of the Jboss configuration (the
> [jboss]/server/[config]-dir which resides in the CVS).
>
> How do we plan on doing this? Of course a Jboss/HSQL combo is not that
> difficult, but if we also want to do WebLogic Oracle, automated installs
> and stuff won't be that easy I think. Do we let it run at one of our
> places or something?

I don't think that the database is that crucial as a variable. At least in
the first instance I would be more concerned about whether changes broke
usage of the MVC framework, AOP, tx management etc, rather than JDBC.

One issue with multiple app servers is that there need to be some
workarounds for individual products. E.g. servlet listeners work fine to
initialize the web application context in Tomcat et al, but don't work in
WebLogic, which runs load on startup servlets such as dispatchers before
listeners. Would we have customization for each server, or test the portable
way?

Yes, I think we would need one or more volunteers to run the tests on their
hardware. This also means we need to decide when to kick it off and how to
report results (emails to all Spring developers, rather than the list?)

Regards,
Rod


> Rod & All,
>
> I think automated integration testing is a great idea.  I would suggest
> that we start with a JBoss/HSQL combo - easy install and the db is
> integrated in the app server.  Later on we could add a WebLogic/Oracle
> combo.  Here we might not want to install the server software for each
> test, but we could create a WebLogic domain and an Oracle
> instance/database using scripts for each test.
>
> Speaking of tests - I have not been able to run the entire test suite
> lately.
> It keeps failing on the
> org.springframework.metadata.bcel.AttributeWriterTests.
>  I can run this test by itself OK under Eclipse, but not using the
> 'tests' target in the build script.  Has anybody else seen this?  Here
> is the error:
> Testsuite: org.springframework.metadata.bcel.AttributeWriterTests
> Tests run: 1, Failures: 1, Errors: 0, Time elapsed: 0.771 sec
>
> Testcase: testAttributes took 0.771 sec
> FAILED
> Expected one custom class attribute expected:<1> but was:<0>
> junit.framework.AssertionFailedError: Expected one custom class
> attribute expected:<1> but was:<0>
> at
> org.springframework.metadata.bcel.AttributeWriterTests.testAttributes(At
> tributeWriterTests.java:74)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
> a:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
> Impl.java:25)
>
> Testcase: testAttributes
>
> Not sure what is going on here.
>
> Thomas
>
> > Hi Guys,
> >
> > I've been thinking about how to improve our testing strategy further.
> >
> > Currently we have excellent--and continually improving--unit test
> > coverage. This is essential and, I believe, the most important form of
>
> > testing for a framework such as Spring.
> >
> > However it would also be good to have automated integration tests that
>
> > help to verify that code changes won't break user applications. This
> > is especially important before releases, but would be really useful
> > possibly as part of a nightly build process. (I don't see any point
> > running unit tests in a nightly build: developers shouldn't commit
> > without doing that.) I think it's important to move beyond ad hoc
> > testing, not just for unit testing.
> >
> > I recently had lunch with Patrick Linskey of SolarMetric (Kodo JDO).
> > He was telling me about their testing, which is impressive. They have
> > multiple machines devoted to continually testing Kodo on different
> > application servers and against different databases. They start from a
>
> > clean install of the app server, then install Kodo and their test
> > applications.
> >
> > Perhaps we could apply a similar approach as we move towards 1.0
> > final.
> >
> > The basic process would probably involve an additional Ant script, and
>
> > look something like this:
> >
> > - be able to do a checkout from a tag or the head
> > - compile and build deployment units
> > - take a clean application server (perhaps tarred) and deploy the
> > spring Jars (and dependencies) from CVS along with the target
> > application from CVS (one of our samples, at least to start with).
> > We'd also need a database set up similarly--probably HSQL or such an
> > embedded database, including create
> > - wait until deployment is complete, unless the target app server can
> tell
> > us
> > - run tests against the app. This might involve HTTP requests with
> something
> > like HTTPUnit. Or maybe there are better tools. If there are any EJB
> > endpoints in any app we could do RMI against them.
> > - clean up so that no temporary files would break the next run
> >
> > There are a number of challenges and a lot of work to get to this
> > point, but I think it would be worth it. I would love to get Spring a
> > reputation for stability and lack of bugs. Tasks would include:
> >
> > - Writing build script
> > - Deciding which app servers to target? I'd suggest Tomcat and JBoss
> > or WebLogic to start off with. Also, where do we keep the app server
> > tarballs? We can't check JBoss or WebLogic into CVS.
> > - Deciding where do these tests run? How often do they run? Who looks
> > at the results? Maybe email would be the best way to go.
> > - Writing test scripts for deployed apps, and deciding on the best
> > tool.
> >
> > Ideally the scripts would allow adding new app servers or sample
> > applications + test scripts, within a consistent framework.
> >
> > I'd like some views on this. Does anyone have experience of doing
> > something similar?
> >
> > Also, a volunteer to take ownersip would be great, assuming we agree
> > it's a good idea! This might be a good opportunity for someone who
> > doesn't presently have CVS access but wants to be a Spring developer,
> > as it wouldn't involve changing existing code. (Not that I want to
> > discourage any present
> > developer.)
> >
> > Regards,
> > Rod
> >
> >
> >
> >
> > -------------------------------------------------------
> > This SF.net email is sponsored by: SF.net Giveback Program.
> > SourceForge.net hosts over 70,000 Open Source Projects. See the people
>
> > who have HELPED US provide better services: Click here:
> > http://sourceforge.net/supporters.php
> > _______________________________________________
> > Springframework-developer mailing list
> > Spr...@li...
> > https://lists.sourceforge.net/lists/listinfo/springframework-developer
> >
>
>
>
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: SF.net Giveback Program.
> SourceForge.net hosts over 70,000 Open Source Projects. See the people
> who have HELPED US provide better services: Click here:
> http://sourceforge.net/supporters.php
> _______________________________________________
> Springframework-developer mailing list
> Spr...@li...
> https://lists.sourceforge.net/lists/listinfo/springframework-developer
>
>

Re: [Springframework-developer] Testing

[Darren D. <da...@da...>]

On Monday 13 October 2003 09:49, Rod Johnson wrote:

> Yes, I think we would need one or more volunteers to run the tests on their
> hardware. This also means we need to decide when to kick it off and how to
> report results (emails to all Spring developers, rather than the list?)

I'd be happy to run the tests on WebSphere 4.x and 5.x servers for you.  Can't 
help with Oracle, but can certainly do MySQL and possibly Sybase if db access 
tests become important later on.



> > > Also, a volunteer to take ownersip would be great, assuming we agree
> > > it's a good idea! This might be a good opportunity for someone who
> > > doesn't presently have CVS access but wants to be a Spring developer,
> > > as it wouldn't involve changing existing code. (Not that I want to
> > > discourage any present
> > > developer.)

if someone better qualified doesn't put their hand up, I'd also be pleased to 
try to take this on too.

Regards,

-- 

Darren Davison
Public Key: http://www.davison.uk.net/key.jsp

Re: [Springframework-developer] Testing

[Rod J. <rod...@in...>]

> if someone better qualified doesn't put their hand up, I'd also be pleased
to
> try to take this on too.

Darren,

If you'd like to take ownership of this that would be great. I think a short
proposal (one page or so of text) would be great so that we can all review
the proposed strategy before getting down to implementation.

Regards,
Rod

[Springframework-developer] App Testing

[Darren D. <da...@da...>]

On Saturday 11 October 2003 18:29, Rod Johnson wrote:

> However it would also be good to have automated integration tests that help
> to verify that code changes won't break user applications. This is
> especially important before releases, but would be really useful possibly
> as part of a nightly build process. (I don't see any point running unit
> tests in a nightly build: developers shouldn't commit without doing that.)
> I think it's important to move beyond ad hoc testing, not just for unit
> testing.

--remainder snipped-- 


as promised a few days ago, I spent a little while thinking about this and 
there are some challenges involved to meet even the primary goals.  I'll 
outline some of my thoughts here, but I'd like to get some feedback from 
people on the list as to what is deemed really important and therefore 
worthwhile spending the time on before expanding much further.

 * The main goal as I understand it is to provide an application testing 
sandbox - ie one that focuses on the usage of Spring and not on the Spring 
classes or components themselves.  In this way we hope to verify that changes 
to Spring which are known not to break compilation or unit testing of Spring 
itself, also can be shown not to break applications depending on it and/or 
maintain backward compatibility.

 * While such tests may be more important as new releases approach, there 
should be no impediment (time and machine resources notwithstanding) to 
performing such tests nightly against CVS snapshots.

 * Tests should be repeatable and the results unambiguous.  This *may* involve 
writing specific new test applications that focus on these goals rather than 
attempting to fit existing sample applications to the purpose which may have 
more of a user-educational benefit, or which are harder to test in a regular 
fashion.

 * Results should be made public in a consistent, user-friendly format, 
possibly via html and with mail notifications to the dev-list.

 * Test applications should be capable of automated deployment and testing on 
various J2EE server platforms.


These are the primary aims as I see them presently.  If anyone has views of 
any kind on the primary aims, I'd be very keen to hear them via the list.  
Following on from these are some other thoughts that are influenced by 
implementation constraints, some of which are outright questions rather than 
suggestions for moving forward.

i. Given that no-one is likely to come forward with a large array of high 
powered hardware for our testing pleasure (but do feel free to correct me), 
the tests will need to be distributable.  We will probably be able to find a 
semi-permanent home for testing Tomcat/JBOSS, but this is the only way we 
will be able to test against many other server platforms - in particular the 
heavyweight and expensive commercial ones.  (SourceForge do have a WebSphere 
box in their compile farm, but I think it may be difficult to run the sort of 
tests we want to run there.)  

ii. With distributable tests running ad-hoc on donated server resources, it 
may not be possible or desirable to operate tests that begin with clean 
platform installs.  In some cases, the platform install may simply be too 
problematic to script anyway, even if time and resource permitted.  How 
critical is this as a requirement?  Whilst certainly feasible on 
Tomcat/JBOSS, I'd suggest that trashing and re-deploying the test apps on an 
existing server install is the best we might get for WebLogic / WebSphere et 
al.

iii. Each application server will require it's own script(s), perhaps even one 
per test application per server.  However, once a test application is 
deployed, the actual test should be designed to be handled in a 
server-agnostic manner unless deliberately testing specific platform 
differences.  Further investigation might show that something like custom ANT 
tasks would be of use somewhere here.

iv. Dependencies need careful consideration in how they may affect test 
results given a widespread test environment.  The database problem has 
already been remarked upon, local dependencies such as db drivers, different 
versions of libraries in the lib/ext (or equivalent) directory of the server 
installation and even possibly different JDK versions are elements that will 
have to be dealt with which wouldn't necessarily be issues in a testing lab.  
It may be possible to report some of this information within the test results 
where sensible values can be retreived.  Does anyone have thoughts on this?

v. Unambiguous results may be a goal too far.  It may be that results will 
have to be verified individually with someone deciding whether a 'failure' is 
really a failure that we need to be concerned with, or just a failure that 
was caused by 'that dodgy db driver again'.



I'll keep my further thoughts to myself for now, I'd be keen to see what the 
feeling is about all of this.

-- 

Darren Davison
Public Key: http://www.davison.uk.net/key.jsp

RE: [Springframework-developer] App Testing

[Alef A. \(JTeam\) <al...@jt...>]

>  * Tests should be repeatable and the results unambiguous.  
> This *may* involve 
> writing specific new test applications that focus on these 
> goals rather than 
> attempting to fit existing sample applications to the purpose 
> which may have 
> more of a user-educational benefit, or which are harder to 
> test in a regular 
> fashion.
Sounds reasonable to me. The goal is to test Spring in production
environments, not to test the sample applications.

>  * Test applications should be capable of automated 
> deployment and testing on 
> various J2EE server platforms.
Well, that sounds handy, otherwise people must be involved every test :)
 
> i. Given that no-one is likely to come forward with a large 
> array of high 
> powered hardware for our testing pleasure (but do feel free 
> to correct me), 
> the tests will need to be distributable.  We will probably be 
> able to find a 
> semi-permanent home for testing Tomcat/JBOSS, but this is the 
> only way we 
> will be able to test against many other server platforms - in 
> particular the 
> heavyweight and expensive commercial ones.  (SourceForge do 
> have a WebSphere 
> box in their compile farm, but I think it may be difficult to 
> run the sort of 
> tests we want to run there.)  
We're running a fully automated build here on three servers (well, you
can hardly call them servers, they're PIII-800s with 512-1024 ram). One
is the CVS server, one is the server building the projects, one of the
dedicated test server. Right now we're testing on a jboss/jetty combo
(tens of automated builds a day), but we've also been able to do
weblogic/oracle combo concurrently. It _is_ going to be a problem to
test WebSphere/9iAS on such machines however... If we need resources,
I'll probably be able to free them...

> ii. With distributable tests running ad-hoc on donated server 
> resources, it 
> may not be possible or desirable to operate tests that begin 
> with clean 
> platform installs.  In some cases, the platform install may 
> simply be too 
> problematic to script anyway, even if time and resource 
> permitted.  How 
> critical is this as a requirement?  Whilst certainly feasible on 
> Tomcat/JBOSS, I'd suggest that trashing and re-deploying the 
> test apps on an 
> existing server install is the best we might get for WebLogic 
> / WebSphere et 
> al.
I don't know how the other servers do it, as far as JBoss is concerned,
the server-configuration directoyr is enough. We could just put that in
the CVS ([jbosshome]/server/default for instance). We're having a
standard jboss-installation on the testserver and we're updating the
configuration directory each build (from the cvs). This works perfeclty.
As I said, I don't know how weblogic/websphere/9iAS are handling this.

> iii. Each application server will require it's own script(s), 
> perhaps even one 
> per test application per server.  However, once a test application is 
> deployed, the actual test should be designed to be handled in a 
> server-agnostic manner unless deliberately testing specific platform 
> differences.  Further investigation might show that something 
> like custom ANT 
> tasks would be of use somewhere here.
Agree...

> iv. Dependencies need careful consideration in how they may 
> affect test 
> results given a widespread test environment.  The database 
> problem has 
> already been remarked upon, local dependencies such as db 
> drivers, different 
> versions of libraries in the lib/ext (or equivalent) 
> directory of the server 
> installation and even possibly different JDK versions are 
> elements that will 
> have to be dealt with which wouldn't necessarily be issues in 
> a testing lab.  
> It may be possible to report some of this information within 
> the test results 
> where sensible values can be retreived.  Does anyone have 
> thoughts on this?
JUnit does this automatically for the VM performing the test, doesn't
it? As far as server configs are involved, are the tests going to move
around across server that much to make it desirable to automate this?

Re: [Springframework-developer] App Testing

[Darren D. <da...@da...>]

On Tuesday 21 October 2003 12:49, Alef Arendsen (JTeam) wrote:

> > iv. Dependencies need careful consideration in how they may
> > affect test
> > results given a widespread test environment.  The database
> > problem has
> > already been remarked upon, local dependencies such as db
> > drivers, different
> > versions of libraries in the lib/ext (or equivalent)
> > directory of the server
> > installation and even possibly different JDK versions are
> > elements that will
> > have to be dealt with which wouldn't necessarily be issues in
> > a testing lab.
> > It may be possible to report some of this information within
> > the test results
> > where sensible values can be retreived.  Does anyone have
> > thoughts on this?
>
> JUnit does this automatically for the VM performing the test, doesn't
> it? As far as server configs are involved, are the tests going to move
> around across server that much to make it desirable to automate this?

What I was thinking was that it's likely that the same application tests 
will be run in totally different environments, dependent on who can donate 
the resources to do it.  What we can't do in such a situation, is isolate 
the application server as the only variable in the test.  If a test passes 
on your server running Tomcat (say) and fails on mine running WebSphere, we 
can't say very much about that failure with any certainty.  You may simply 
have had a slightly different version of some dependency that we're unable 
to ship as part of the test but which still influenced it.

If it were possible to have the tests verify version information for major 
components and report them as part of the results, we may later see 
patterns emerging in the failures.  On the other hand, it may be nothing of 
an issue if we can indeed include all the major components (JDK and J2EE 
server aside of course) for whatever tests we're running.  I suspect for 
the most part we will be able to.

-- 

Darren Davison
Public Key: http://www.davison.uk.net/key.jsp

Re: [Springframework-developer] App Testing

[Rod J. <rod...@in...>]

Darren

>  * The main goal as I understand it is to provide an application testing
> sandbox - ie one that focuses on the usage of Spring and not on the Spring
> classes or components themselves.  In this way we hope to verify that
changes
> to Spring which are known not to break compilation or unit testing of
Spring
> itself, also can be shown not to break applications depending on it and/or
> maintain backward compatibility.

Yes

>  * While such tests may be more important as new releases approach, there
> should be no impediment (time and machine resources notwithstanding) to
> performing such tests nightly against CVS snapshots.

Yes

>  * Tests should be repeatable and the results unambiguous.  This *may*
involve
> writing specific new test applications that focus on these goals rather
than
> attempting to fit existing sample applications to the purpose which may
have
> more of a user-educational benefit, or which are harder to test in a
regular
> fashion.

Yes, but it should be possible to add more apps to the suite as necessary.

>  * Results should be made public in a consistent, user-friendly format,
> possibly via html and with mail notifications to the dev-list.

Yes. Simple success or failure, and information about any failure.

>
>  * Test applications should be capable of automated deployment and testing
on
> various J2EE server platforms.

Yes.


> i. Given that no-one is likely to come forward with a large array of high
> powered hardware for our testing pleasure (but do feel free to correct
me),
> the tests will need to be distributable.  We will probably be able to find
a
> semi-permanent home for testing Tomcat/JBOSS, but this is the only way we
> will be able to test against many other server platforms - in particular
the
> heavyweight and expensive commercial ones.  (SourceForge do have a
WebSphere
> box in their compile farm, but I think it may be difficult to run the sort
of
> tests we want to run there.)
>
> ii. With distributable tests running ad-hoc on donated server resources,
it
> may not be possible or desirable to operate tests that begin with clean
> platform installs.  In some cases, the platform install may simply be too
> problematic to script anyway, even if time and resource permitted.  How
> critical is this as a requirement?  Whilst certainly feasible on
> Tomcat/JBOSS, I'd suggest that trashing and re-deploying the test apps on
an
> existing server install is the best we might get for WebLogic / WebSphere
et
> al.

I agree. With JBoss/Tomcat I'd unzip it. Also Orion if we wanted to. I don't
know much about WebSphere configuration. WebLogic has nice clean separate
server config directories, so we should be able to handle it at that level.

> iii. Each application server will require it's own script(s), perhaps even
one
> per test application per server.  However, once a test application is
> deployed, the actual test should be designed to be handled in a
> server-agnostic manner unless deliberately testing specific platform
> differences.  Further investigation might show that something like custom
ANT
> tasks would be of use somewhere here.

Good point. We need to choose a test tool. RMI is handy for EJB-based tests.
Other stuff may need to be web tested via Latka or the like. Or we could
deploy some test cases into the server. I'm not a big fan of things like
Cactus in testing applications, but they may make sense for this kind of
thing. We could have some tests outside any container of course. Like
checking transactional behaviour via AOP against a database, and checking
the results.

> iv. Dependencies need careful consideration in how they may affect test
> results given a widespread test environment.  The database problem has
> already been remarked upon, local dependencies such as db drivers,
different
> versions of libraries in the lib/ext (or equivalent) directory of the
server
> installation and even possibly different JDK versions are elements that
will
> have to be dealt with which wouldn't necessarily be issues in a testing
lab.
> It may be possible to report some of this information within the test
results
> where sensible values can be retreived.  Does anyone have thoughts on
this?

We should be able to say which versions of a server we've tested on. E.g.
WebLogic 8.1 SP1, JBoss 3.x.x. We can't guarantee users testing on every
combination out there, but we should be able to say "that problem doesn't
occur with certain combinations we've tested."

>
> v. Unambiguous results may be a goal too far.  It may be that results will
> have to be verified individually with someone deciding whether a 'failure'
is
> really a failure that we need to be concerned with, or just a failure that
> was caused by 'that dodgy db driver again'.

SolarMetric have had massive issues with this in setting up their test
suites. There are a lot of buggy JDBC drivers out there when you start to
push them hard. However, I don't think we need to aim to test every database
out there. I'd be happy just to use an embedded database to start off.
Greatly simplifies the config process also.

I think a first goal might be Tomcat + HSQL. Easy to do the "just unzip"
approach and meets the basic goal of verifying that we didn't break
everyone's web app in changes today.

The goal of saying "Spring works end to end on app server X" is a separate,
but also important, goal, I think.

Regards,
Rod

Re: [Springframework-developer] App Testing

[Darren D. <da...@da...>]

On Wednesday 22 October 2003 14:18, Rod Johnson wrote:

> I agree. With JBoss/Tomcat I'd unzip it. Also Orion if we wanted to. I
> don't know much about WebSphere configuration. WebLogic has nice clean
> separate server config directories, so we should be able to handle it at
> that level.

WebSphere will probably prove too tricky.  It requires an RDBMS to store its 
config per WebSphere Domain (cluster) and will install DB2 for the purpose 
if you don't have something already available for the job.  It's not beyond 
the realms of possibility to script it all, but it's a greatly diminished 
return for the effort.

> I think a first goal might be Tomcat + HSQL. Easy to do the "just unzip"
> approach and meets the basic goal of verifying that we didn't break
> everyone's web app in changes today.

I'll start along these lines.

Regards,


-- 

Darren Davison
Public Key: http://www.davison.uk.net/key.jsp