[Asunit-users] Unit testing / Integration testing

[<ma...@de...>]

Hi Alias,

Extremely well said. You have mirrored many of my own, and I'm sure many other's, thoughts. I'm not sure about the hows, but the whys are definitely all there.

Perhaps the first step could be a change of definition/mind set. How about 'Integration Tests' instead of 'Unit Tests' for what you're talking about. Yeah, I know. Same problem, different name, but there are a lot of presumptions and expectations attached to a name, especially something like 'Unit Test'. After all, a unit test should be a unit test. That's what it was designed for. That's what it's good at.

Anyway, enough rambling. I would definitely be interested in discussing this point more with anyone interested.

Hope you're all well,

Marcus.

ma...@de...

-----Original Message-----
From: asu...@li... [mailto:asu...@li...] On Behalf Of asu...@li...
Sent: Saturday, January 07, 2006 1:24 PM
To: asu...@li...
Subject: Asunit-users digest, Vol 1 #110 - 1 msg

Send Asunit-users mailing list submissions to
	asu...@li...

To subscribe or unsubscribe via the World Wide Web, visit
	https://lists.sourceforge.net/lists/listinfo/asunit-users
or, via email, send a message with subject or body 'help' to
	asu...@li...

You can reach the person managing the list at
	asu...@li...

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Asunit-users digest..."


Today's Topics:

   1. Re: Re: Testing asynchronous functions (Alias)

--__--__--

Message: 1
Date: Fri, 6 Jan 2006 14:23:31 +0000
From: Alias <ali...@gm...>
To: asu...@li...
Subject: Re: [Asunit-users] Re: Testing asynchronous functions
Reply-To: asu...@li...

Hi Luke,

Sorry for the delay, I've been really busy leading up to the holidays
and my Gmail is so flooded with mailing lists I sometimes miss
messages.

I'm thinking that EventDrivenTestCase is the way to go. What we need
is a testcase that will basically listen to events - that means that
the teardown & setup stuff would have to be called every time.

I know what you're saying about testing too much. The thing though, is
that I think there's a fundamental limit to unit testing which makes
it less useful than it could be. This (admittedly angry, ranty and
perhaps a little over the top) article begins to touch on an important
point:
http://www.pyrasun.com/mike/mt/archives/2004/07/11/15.41.04/index.html

Essentially, unit tests are great, but where they fall down is their
inability to deal with interactions between different parts of the
system. Ideally, I'd like to write my tests, and then run my
application as intended, with the unit tests running as well, so that
I can actually test the entire system as it runs. Each test case can
be run once at the start to test each class in isolation, but lots of
tests need to be run in some kind of production environment. I guess
there is a line of distinction to be drawn between exception handling,
acceptance tests and unit testing, but for me the unit testing would
be much more useful if I could write it in conjunction with the UI
code and then compile, click stuff, and if unit tests fail then give
me exact info about which tests have failed and why. Sometimes it just
doesn't make sense for a class to be tested completely in isolation -
some problems only become apparent when they are used in conjunction,
even though all the tests are passed.

Unit testing for individual classes has been very useful to me, and
I'd like to extend that level of coverage to the UI, as far as
possible - I know it's pretty much impossible to test human
interaction stuff completely accurately, but it must surely be
possible to test stuff in a step by step manner.

However - I'm beginning to think a different methodology is needed for
the kind of testing I'm  thinking of. Here are my problems with unit
testing:

    * Unit testing doesn't really make sense for UI development - I
won't go into why this is, because it's pretty self apparent
    * Unit testing only tests such small pieces of functionality that
it only really useful in the context of single, complex classes which
have simple, defined inputs & output and minimal interaction with
other classes
    * Unit testing can't easily be used to test asynchronous functions
    * It is difficult, if not impossible, to unit test every piece of
functionality in your classes while they are interacting in a
production environment
    * It is entirely possible (and perhaps likely) that an application
could be totally broken and badly engineered, despite passing a huge
number of unit tests

This is not to say that unit testing is useless or a waste of time.
However, in it's current form, its use in UI development is severely
limited.Let's look at some of the goals would expect from a perfect
testing system:

    * Test code must be completely seperate from production code, so
that when I am confident that my application works, I can remove the
test harness & be reasonably sure that the application will still work
reliably - unit testing accomplishes this very well.
    * I need to be able to test an application as it is running - that
is, in it's production form, in a live environment, doing actual,
specific tasks - not just passing tests that I myself have designed
    * I would like to produce detailed feedback and logs from the
tests themselves, rather than having a seperate logging process

Although it is possible to accomplish many of these things in ASUnit,
it's difficult to do so in the framework as it stands. Hence, I think
that it's probably the case that these problems are more to do with
the very concept of unit testing itself, rather than any specific
implementation.

What I'm thinking of, as a solution, is this - what I need is the
ability to call some kind of seperate test functionality from inside
my production code. For example:

//I've just recieved a response from an AMF gateway -
var myAmfResult:Object =3D response;
trace(RuntimeTest.runTest("testAmfResponseIsValid",this,myAmfResult));


My ContinousTest class would then run the appropriate test on the live
instance (passed as a parameter to the test) and from then on, you're
in similar territory to unit tests, except you don't have the
teardown/setup functionality - this is because you wouldn't have this
in production either, and you can't rely on always having a fresh
instance of something to test on. Ideally, I'd like to combine my
runtime tests with the unit tests, (maybe in the same class file?) so
that the entire testing system can be removed without leaving any test
code in the application.

This saves the work that would be otherwise consumed in creating
complex testcases & mocks, essentially working around the limitations
of the unit test framework.

Note the use of the trace() function - this is deliberate, becasue we
can use the "remove trace actions" compiler directive to detach the
runtime test. Each call to RuntimeTest could return the test results,
or simply an empty string and pass the output to a logger if we didn't
want it to clutter up the output panel.

Does this make sense? What do you guys think?

Cheers,
Alias









On 12/8/05, Luke Bayes <lb...@gm...> wrote:
> Hey Rob,
>
> Sorry about the delayed response on my part. I've been talking with Ali
> about this issue and giving it some thought myself.
>
> What I'm hearing is that you need a single TestCase that will execute som=
e
> action, pause for an arbitrary amount of time, and then perform assertion=
s
> around each test method, rather than once for a test case.
>
> My first guess is that this should be possible by overriding the runBare
> method of TestCase. If you're working in AS 3, this might have a larger
> impact because of how we implemented test failures. In AS 2, failures wer=
e
> simply transmitted to the Runner from anyone that wanted to pass them, in
> the AS 3 build, failures are transmited by "throwing" a TestFailure objec=
t
> up the call stack (This is closer to the way JUnit works). This may cause
> some problems if the called methods are no longer in the same call stack
> that the runner initiated... It might still work fine - I'm just not sure=
.
> At a minimum, there should be a method in TestCase or Assert that allows =
us
> to transmit a failure directly - the overriden runBare could then pass
> failures up to that method...
>
> At the end of the day, it still feels like you're trying to test too much=
 at
> once. If you're trying to test features of a preloader, there should be a
> way to do that using the AsynchronousTestCase as provided. But I would
> encourage you to decouple the testing of the preloader functionality from
> the testing of the features being loaded. Admittedly, this can be a compl=
ex
> problem and I believe even java developers still struggle with it.
>
> If you're interested in getting into the sources, please let me know as t=
he
> CVS repository on sourceforge is no longer up to date. It was getting rea=
lly
> messy in there so we moved over to Subversion.
>
> Also, if you're wanting to add these features, let's come up with a
> different name since AsynchronousTestCase is already being used for the
> feature that allows pausing prior to the test case execution. Maybe
> PausableTestCase? EventDrivenTestCase? IntervalDrivenTestCase? Or perhaps=
,
> we should just refactor these features into the existing
> AsynchronousTestCase?
>
> Another possible alternative is that you could simply use what is already
> there and implement a unique test case class for each asynchronous test t=
hat
> you want to verify...
>
> Let me know what you think -
>
>
>
> Luke Bayes
> www.asunit.org
>
>
>
>
>
>
>
>
>
>



--__--__--

_______________________________________________
Asunit-users mailing list
Asu...@li...
https://lists.sourceforge.net/lists/listinfo/asunit-users


End of Asunit-users Digest