Re: [openMSX-devel] Analysis of bugs fixed in 0.10.1

[Wouter V. <ver...@gm...>]

This is mostly an "I agree" reply, with maybe some different nuances.


On Wed, May 7, 2014 at 7:04 AM, Maarten ter Huurne <ma...@tr...>
wrote:
> On Tuesday 06 May 2014 11:29:03 Laurens Holst wrote:
>
> > I think given that in a previous mail there was the complaint that
> > releases aren’t happening often enough because the release process is
> > tedious, the last thing you want to do is to create a long elaborate
> > manual checklist.
>
> Agreed.

Agreed.


> About half of the bugs fixed in 0.10.1 would have been likely to be found
> with unit testing, and that is if there was test coverage of those areas
of
> the code, of course. So it's not a solution to everything, but it is the
> most effective of the testing strategies I looked at.

I mostly agree. I don't believe unittesting can find all or even most of the
recent bugs, especially in the short term. But it certainly seems like the
most realistic solution.


> > Personally, it is one of the first things I set up when I start working
> > on a project if it doesn’t have them already (more often than not).
> > Having introduced unit testing to a number of projects, I think you
> > shouldn’t aim for covering the whole openMSX code with unit tests
> > straight away.
>
> We have over 170k lines of code in openMSX, so I don't think full coverage
> is feasible even if we wanted to, certainly not in the short to mid term.
> ...
> > Regarding the need to refactor before you can start unit testing; it is
> > true that singletons make code harder to unit test, but for every thing
> > that is hard to unit test in the current code, there will be 10 things
> > that are easy to unit test.
> ...
> I'm not sure that 1:10 ratio will be true for openMSX: without either
> dependecy reduction or useful mocks for classes like MSXMotherBoard,
> Reactor, CliComm etc. there won't be much code that can be tested.

My personal (rather limited) experience with unit tests is that it's
easy to setup for small/new projects but hard for big/old projects. Code
should be designed for testability.

I'm afraid that the ratio 1:10 will be more like 1:1 for openMSX. For
example the code in src/utils will be easy to unittest, but unfortunately
that's not the code where most bugs are located.

But even though I think it will be a lot harder than what Laurens thinks,
I still think we should try it. And at the very least have the 'easy' parts
unittested.


> > Note you don’t need to take the strictest
> > definition of unit testing, where you mock every object that is not
> > local to the test (I surely never do that). If unit tests run
> > sequentially, you can set-up global state for them as needed. As you’re
> > adding new features or fixing bugs and writing unit tests for them, you
> > can take the opportunity to gradually refactor some code to better
> > support testing.

Since you seem to have quite some experience with unittesting, maybe
you could help us out. E.g. show us how some of our more difficult to test
code can be made to fit in a unittest framework? I'd be very happy to be
proven wrong about that 1:1 ratio ;-)


> Since asserts will end a process, it is good to have each test case in a
> separate process anyway, otherwise an error on one test case would prevent
> other test cases from running.

(detail) I've seen frameworks that redefine the assert macro (I don't
remember which ones though). But separate processes is an OK solution
as well.


> All that said, unit testing does require people to write unit tests. Is
> anyone willing to write tests for parts of openMSX? If so, I can
integrate a
> unit test framework into the build.

I volunteer for the easy stuff (e.g. src/utils), but for the more
interesting
stuff (like the MSXDevice classes) I'd like to have some guidance initially.


An important first step would be to add *some* kind of framework in the
build
system. It doesn't even really matter which framework, anything is better
than
what we have now (= none at all). IOW I'd like to avoid a long discussion
about
what/how the framework should be/work. Just add something and improve upon
it
over time (or even completely switch framework).


Wouter