Re: [Asunit-users] Unit testing / Integration testing

[Alias <ali...@gm...>]

Hi Luke,

Thanks for your mail.

> - In a desktop application, one has much more freedom in terms of file si=
ze,
> but the swf apps that we have developed tend to be very sensitive to even=
 a
> few kilobytes of inflation. I believe the non-visual AsUnit 2.x framework=
 is
> 7KB to 14KB right now.
>
> - The SWF runtime is relatively slow in terms of memory and CPU usage. I
> haven't yet encountered a complex SWF application that left much in terms=
 of
> system resources to spare - especially during periods of heavy activity, =
the
> very times when you would most likely encounter failures.
>

Both fair points, no argument there.

> - Before you go too far, you should make absolutely certain that compilin=
g
> with "trace" turned off, actually omits objects (other than string
> primitives) that were referenced in trace statements from the SWF. I susp=
ect
> that the only way to remove trace statements is to perform the removal
> "after" the source has been tokenized. I could certainly be wrong about
> this, but it is possible that the compiler does not re-run dependency
> analysis after trace removal. If this is the case, wrapping these referen=
ces
> in trace statements probably would not work.
>

That's an excellent suggestion, I'll look into that. My impression was
that it was a precompile directive, but now I think about it, that's a
totally unfounded assumption.

> - Finally - and most importantly, there may be licensing issues associate=
d
> with including AsUnit inside of a commercial product. I'm not a lawyer, s=
o I
> could be wrong on this issue, but the way that I understand the LGPL, is
> that if one ships a library covered by this license (as AsUnit is), it mu=
st
> be compiled separately from the associated product - and linked
> "dynamically" at runtime. This works fine for a test framework, because y=
ou
> don't actually "ship" the test harness as part of the product. It is (tod=
ay)
> compiled separately. This issue will probably prevent most projects from
> being able to include AsUnit at runtime. On this topic - if this idea gai=
ns
> traction - we will definitely consider moving to a different license in t=
he
> future - but I'm unclear about what freedom we really have in this issue =
(at
> least for AsUnit 2.x)...
>
> Overall, I definitely support the idea of extending AsUnit to work in the
> context of integration (or UI) tests, but I'm not convinced that we shoul=
d
> do much work with the idea that people are going to ship AsUnit in releas=
ed,
> production code.
>

Ok - I wouldn't be proposing to create a situation whereby a project
becomes dependant on, or needs to ship with, AsUnit. The integration
testing stuff would be another part of the test harness, or would
allow you to easily connect the asserts to your own debug/reporting
code. There are three different user stories here:

  1) I want to do test driven development, and then do functional
testing as I go, and know if I break something.

  2) I want to have accurate debug information when my code breaks

  3) I want to ship the product with this debug information to the
client, so that I can get accurate bug reports

So, it's important to seperate these concerns. The first two are my
main concern -  the third one has come out of the process of figuring
out how to do the first two. Shippping with asserts on is a totally
seperate issue to having a more reliable test harness - I wouldn't
want to create a system which forces the user to do one or the other.
However, as the owners of the project, the licencing is absolutely
your decision.

> - I think we should also consider reliability requirements. Different
> projects can have wide variance in terms how reliable they must be. For
> example, it is catastrophic when there is a software failure in the space
> shuttle or a pace maker. I'm not sure that most web apps actually have th=
e
> same kind of uptime requirements. I'm not saying that failures should be
> taken lightly - I'm just saying that one needs to evaluate the cost of th=
at
> security and make appropriate trade offs.

That's also an excellent point. Do you know of any formal measurements
for reliability requirements? Say,
financial/military/consumer/experimental?

Let me know,
Alias