Regression Tests

Regression Tests

Before committing any changes to the SVN repository, you should run all regression tests and make sure no new tests fail. Our testing framework is made up of several shell scripts. There are testsuite directories in several subdirectories to test different programs in the gputils suite.

Microchip Tests

Microchip donated a set of tests that we keep in the gpasm/testsuite/gpasm.mchip subdirectory.

Framework

We're currently using shell scripts for our testing framework. The tests work in a very structured way, taking files from one directory, processing them, and comparing the results to files kept in other directories. If the program under test fails, the whole test run fails. Otherwise, they only warn about any differences from the expected output.

For example, under gpasm.project and gpasm.mchip, any asm file under asmfiles will be assembled in absolute mode, and the resulting hex file will be compared to the file of the same name in the hexfiles dir. An asm file under objasm will be assembled in relocatable mode, and the resulting object file will be compared to an object file in objfiles.

Craig Franklin had talked about moving the testing to dejagnu. Since the tests are so structured, it should be pretty simple to port them to any framework. We're not particular about what testing framework we use, but it would be handy to do something intelligent with the tests that don't yet pass so that we know right away if any of them pass.

Bugs

There are currently two real problems with the testing system. These aren't in the bug tracking system because they're not user-visible bugs (they only affect developers).

Dead gpasm.mchip Tests

We have some gpasm.mchip tests that have never passed because of minor but stubborn bugs in gpasm. It's not very visible when any of these tests fail, so to detect a regression right now you have to run the tests twice and diff the results. We need to make it very visible when a working test fails or a non-working test passes, so we need to flag each of the tests that have never passed somehow. The simplest solution with the current framework would be to create a gpasm.mchip.notworking and write the test function to ignore when these tests fail but warn when one of them passes (so we can move it into the regular regression tests).

bash/sh

They depend on bash, but claim to work in sh (#!/bin/sh at the top). This gives errors when you run them on, for example, a fresh Debian or Ubuntu system because they use a barebones sh version (dash) that doesn't support bash features.