Re: [Rapideuphoria-develop] EuTest

[Jim C. B. <jbr...@li...>]

On Fri, May 28, 2010 at 09:54:29AM -0400, Matt Lewis wrote:
> > I thought we already had this.
> 
> Had what?  Not sure what you're asking.

The ability to detect, with eutest, the issues I've described below...

> > This is equally important (not less important), and it is functionality
> > that we need to have. If a test for an out-of-bounds index is failing
> > due to a machine level exception, we need to know about it before a
> > release comes out!
> 
> Yes, we do.  And to my knowledge, this hasn't happened.

So?

> The biggest
> issue we've had is with keeping up with error message changes.
> 
> There were some issues
> recently that were basically about making the error files cross
> platform compatible.  I think a lot of it had to do with path issues.
> I haven't really worked a lot with those, though.

These are less important, but still worth fixing.

> 
> I couldn't really follow Shawn's example very well (aside from the
> fact that he claims to have found and fixed a bug)

If I understand correctly, he claims to have found a bug in the ability
that I thought eutest already had, and to have fixed it.

That is, he found a bug in eutest where it was not correctly checking
the error message to see if it was the right error message (or a
different, unexpected error message such as a machine exception) before
issuing a success or failure.

A bug in parsing or comparision of control.err or ex.err, I guess.

> so I may be
> missing the bigger picture here.
> 

I think it ties into what Shawn originally said: that Euphoria itself is
not a good test of eutest's capabilities.

We can't simply make changes to eutest and then run the Euphoria
unittests to know if something has been broken. We either need a better
way to check for regressions in eutest or (what I believe Shawn is
advocating) declare eutest stable and feature-complete, in which case we
never need to make changes to eutest's code and thus never have to worry
about testing for regressions or new bugs.

> > Do we have any regression testing done on eutest at all? I didn't think
> > so but I haven't been following the code that closely.
> 
> I guess...to the extent that people run it and notice issues.  There
> are no unit tests for eutest that I am aware of.

I suppose it's a good thing that the main author of eutest considers the
product to be stable and feature-complete, then.

> 
> -- 
> Matt Lewis
> 
> ------------------------------------------------------------------------------
> 
> _______________________________________________
> Rapideuphoria-develop mailing list
> Rap...@li...
> https://lists.sourceforge.net/lists/listinfo/rapideuphoria-develop

-- 
Infinite complexity begets infinite beauty.
Infinite precision begets infinite perfection.