Re: [GD-General] problems while debugger is attached

[Chris C. <can...@gm...>]

There was a thread on one of the dev lists a while back about test
driven development, and a lot of arguing back and forth about exactly
what constituted a unit test versus a system test. I sort of felt it
missed the point entirely, which is just to test the code in ways that
might break! I treat test driven development as a way of thinking,
where while (or preferrably before) you are developing code, you also
write tests to exercise your code and make sure it does what you'd
expect. Multi-threaded code, unless written very tightly and cleanly,
is only testable at the functional level.

I try to make my tests as cruel as possible. In the case of
multi-threaded bits of code, I have taken to writing tests which
substitute the lock/unlock classes for test versions which introduce
random but non-trivial delays before and after locks. Then I take a
section of the system and run it through its paces, multiple times.

Solid multi-threaded code should never fail, or do anything
unexpected, even in the presence of massive delays on any particular
part of the code. The worst that should happen is that the system
slows to a crawl, but the output should still be sensible.

For example, we had a GUI thread, which dealt with overlaying
informational display over a 3D system (NB: this wasn't in games, I've
yet to work for a games company with a decent TDD setup). It was
interleaved with rendering, but disconnected in terms of update
frequency, and also had to be interleaved with data access from an
underlying system. So I replaced the underlying system and renderer
threads with a mock version which simply locked and unlocked as if it
were doing work, but actually did nothing, and subbed in the test
locking class. Then the test ran through a pre-determined set of
operations. The randomness was seeded, so deterministic; however you
couldn't predict what would be happening when. The test got repeated
10 times without re-seeding the randomness.

Admittedly, it only found one bug the whole time I used it (deep in
one of the GUI sub-classes there was an else case which didn't free a
locked resource), but it was a bug which would only have shown up in
the real system (it would only get hit if the lock was held by
something else mid-way through the function being called).

All of this is just an extension of how I took to debugging live race
conditions - by introducing random length lock/unlock delays.
Obviously in a live build thats a real pain in the neck - imagine
trying to play a game where the framerate spiked massively and
randomly, in an effort to repro a bug. When I did do that, I ended up
tying it to a debug button combo toggling a flag to make the locks
start misbehaving. In most cases, as soon as you set the random delay
flag, a few frames later the build would hang, and I could go look at
the locking logs to see which resource was deadlocked, and what the
history was. TDD gives you a way to do the same test, but without the
unpredictability of recreation steps.

ChrisC
Black Company Studios

On 23/07/06, Noel Llopis <ll...@co...> wrote:
> On 7/23/06, Chris Chapman <can...@gm...> wrote:
> > Before I started using test driven development for these things, I
> > could spend ages beating my head against weird behaviour like that.
>
> Chris,
>
> I'm curious about how test-driven development helped with situations
> like that. Is it because you simply don't use the debugger as much, or
> because it helps avoid those race conditions somehow?
>
> When I apply test-driven development it's at a very small scale
> (function or class), so none of the tests actually deal with multiple
> threads, inter-process communication, etc (the functional tests do
> though). So that's one situation where I haven't found TDD to help
> much with and I was very curious about your comment.
>
>
> --Noel
>