Regression Tests

This page lists things that should be checked after any major changes to a low-level system (Mesh, CollisionDetection?, etc.) or before a release.

Memory Verification

1. Checked iterators: Ensure that doing some basic operations (loading a map, shooting, etc.) works without any aborts while using checked iterators. It might not be a bad idea to do most of the following regression testing with checked iterators enabled as it should exercise the code fairly well.
2. Valgrind: Probably not something to be run every release, but periodically (especially if there are stability problems, of course) it wouldn't hurt to do a run under Valgrind. Of course, that has significant performance impacts so this testing will have to be much more limited.

Collision Detection

1. Clipping through the ground: In the past it has been possible to clip through the ground and other objects under certain circumstances. The bots are now smart enough to win rounds, so you can't just let them walk around to test this any more. Or at least code changes will be required to do so. However, this hasn't been an issue in quite a while so it's probably okay to just skip it unless you've made changes that might cause it to reoccur.

2. Splash damage to large tris: Due to some changes in the collision detection code, it was possible to shoot a large triangle in the middle with a weapon that does splash damage and have that damage not register properly. A good way to test this is to load the map tears and walk to the cliff above the bottom base. Shooting the base on its large, flat top should register a hit.

3. Shooting individual body parts: Frequently after major changes individual body part damage has stopped working, either because collisions weren't detected at all, or they were detected improperly. To test this ensure that it is possible to destroy a single arm from a stationary bot or test player from a second computer. Bots can be told to stay still by setting the botsmove cvar to 0.

Mesh

1. Mesh has had a number of problems after major changes to the rendering code. As I remember what those were I will add them here. Hopefully since the Mesh refactoring this won't be such an issue.

Particles

1. Sight: The sight particle is unique because it's TTL is 0, so it frequently misbehaves. Make sure that it is a single solid laser with a single solid dot on whatever it hits, and that there is no ghosting when moving side to side. This can pop up even after completely unrelated changes.
2. Tracers: The smoke tracer for rockets is an animated particle, which makes it somewhat unique, so occasionally it gets messed up by changes.
3. Explosions: The explosion particle used to be a problem, but it shouldn't be anymore.

General Rendering

1. Validate shaders: Because NVidia's GLSL compiler is rather forgiving of mistakes and I develop on an NVidia card, bugs can occasionally sneak into the shaders. I have a script that will automatically run the 3DLabs GLSL Validator on all of the shaders to (hopefully) ensure that they will work on all video cards.

Gameplay

1. Game Over: Ensure that one team or the other winning and causing the server to cycle to the next map works correctly. This is not so much a problem area as something that doesn't come up frequently during normal testing so it needs to be explicitly done from time to time. Edit: Actually that's not true. After adding bots it was also a problem area, so make sure you do this. :-)

2. Bot AI: If you've made bot changes or map changes, test that they haven't broken the bots on any of the maps where they're expected to work. The only good way to test this is to play against them and watch for incorrect behavior.

Release Packaging

1. Shadow the user: After building the archive files for each platform, install them on a clean system and run the game as a sanity check of the packaging system. If a clean system is not available (as will often be the case), at least install the files to a separate location and temporarily rename the .coldest directory to ensure that the base install works.
2. Library check: After appropriately setting LD_LIBRARY_PATH=lib, run ldd on the binary and make sure that the expected libraries are being loaded from the lib directory and not the system libs. This should only be a problem when a new library has been added.

Spectating and Replays

1. Spectating: Ensure that spectating still works as expected
2. Replays: Load a replay (some sort of standard replay should be created) to make sure that no unintended incompatibilities were introduced. If a backwards compatibility breaking change was made intentionally then this doesn't apply of course. That should be avoided if possible however.
3. Recording: Record a replay and play it back to check that this functionality still works.