written by Borut Ražem, based on and partially copied from the Proposed Test Suite Design document, written by Michael Hope
Sdcc regression test suite is the main quality assurance (QA) mechanism used by the sdcc project. The regression test results indicate the quality of the sdcc compiler build. Regression tests are designed in the way that they pass in case the sdcc compiler works correctly.
Auto-magically executed regression tests
Sdcc regression tests are auto-magically executed on a daily basis on sdcc snapshot builds. They are executed on the sdcc distributed compile farm (DCF) servers on different platforms. The results are shown in the RT column on the SDCC - Snapshot Builds web page. They can be analyzed on the SDCC - Regression Tests web page.
Manual execution of regression tests
Regression tests should also be run locally on the developers machine by the developer implementing new functionality or fixing a bug before committing changes to the sdcc subversion repository.
How to execute them manually?
Executing the complete test suite:
cd support/regression make
Execute regression tests for one target:
cd support/regression make test-<target>
Re-execute only one test case:
cd support/regression touch tests/<test-case>.c make
Since the tests must pass compilation and linkage it is impossible to test proper generation of error messages.
And since the output to stderr is not parsed either it is also impossible to check for warnings.
Sdcc regression testing consists of the following steps preformed by support/regression/Makefile:
m4 macro preprocessor:
python generate-cases.py preprocessor:
sdcc compiler / assembler / linker:
simulator (ucsim, gpsim, ...):
python get_ticks.py script:
extracts the size and ticks from the .sim files in support/regression/gen/<target>/<test>/ and appends this info to the .out files in support/regression/gen/<target>/<test>/.</test></target></test></target>
all the .out files from support/regression/gen/<target></target>/<test>/ are combined into support/regression/results/<target></target></<test>.out></test>by the make process.
python compact-results.py sript:
python collate-results.py script:
M4 macro preprocessor
The regression test source files with extension .m4 are first preprocessed by the m4 macro preprocessor. The m4 macros implemented in file support/regression/m4include/rtmacros.m4 are included in preprocessing.
The following macros are currently implemented:
forloop(var, from, to, stmt) foreachq(x, `item_1, item_2, ..., item_n', stmt) foreach(x, (item_1, item_2, ..., item_n), stmt)
An example
foreach(`int_right', (0x0080, 0x0100, 0x0200, 0x0400, 0x0800, 0x1000, 0x2000, 0x4000, 0x8000), ` int lwr_if_`'int_right (unsigned left) { if (left < int_right) return 1; else return 0; } int lwr_`'int_right (unsigned left) { return left < int_right; } void test_lwr_`'int_right (void) { ASSERT (lwr_if_`'int_right (int_right - 1)); ASSERT (!lwr_if_`'int_right (int_right)); ASSERT (!lwr_if_`'int_right (int_right + 1)); ASSERT (lwr_`'int_right (int_right - 1)); ASSERT (!lwr_`'int_right (int_right)); ASSERT (!lwr_`'int_right (int_right + 1)); } ')dnl
Python generate-cases.py preprocessor
Regression test source files are not complete c programs: they don't include the main()
function. The main()
function is generated by the generate-cases.py preprocessor. It includes calls to all functions in the regression test source file, whose name begins with test
, for example testMyFunctionality()
or test_bugXXXX(void)
. The test
functions should be of type void
and without parameters.
The generate-cases.py also generates permutations using metadata in the source file. Meta data includes permutation information, exception information, and permutation exceptions.
Meta data shall be global to the file. Meta data names consist of the lower case alphanumerics. Test case specific meta data (fields) shall be stored in a comment block at the start of the file. This is only due to style.
A field definition shall consist of
The values shall be stripped of leading and trailing white space.
Permutation exceptions are by port only. Exceptions to a field are specified by a modified field definition. An exception definition consists of
An instance of the test case shall be generated for each permutation of the test case specific meta data fields.
The runtime meta fields are
Most of the runtime fields are not very usable. They are there for completeness.
Meta fields may be accessed inside the test case by enclosing them in curly brackets. The curly brackets will be interpreted anywhere inside the test case, including inside quoted strings. Field names that are not recognized will be passed through including the brackets. Note that it is therefore impossible to use some strings within the test case.
Test case function names should include the permuted fields in the name to reduce name collisions.
An example
The following code generates a simple increment test for all combinations of the storage classes and all combinations of the data sizes. This is a bad example as the optimizer will often remove most of this code. Note the comma after class static
which permutes into an empty string. The corresponding filename will use none
for the permutation.
/** Test for increment. type: char, int, long Z80 port does not fully support longs (4 byte) type[z80]: char, int class: register, static, */ static void testInc{class}{types}(void) { {class} {type} i = 0; i = i + 1; ASSERT((i == 1)); }
When designing a regression test, the designer has to make a compromise between:
The regression test case should test the generated code as much as possible by keeping the testing time as short as possible.
In case of bug fixing usually the minimal code that was used to reproduce the bug is included into the regression test suite.
In the case of new implementation the new functionality and critical corner cases are tested.
M4 macro preprocessor vs. python generate-cases.py preprocessor
The regression tests often generate different code for handling different types (Byte, Word, DWord, and the signed forms). Meta information could be used to permute the different test cases across the different types and other items.
The python generate-cases.py preprocessor generates one file per each permutation. This means that if the regression test source file includes some tests which don't depend on permuted items, the same tests are executed number of permutations times. Each permutation source file is compiled and executed / simulated independently, which means an overhead in compilation and execution / simulation.
Using the m4 macro foreach
or foreachq
directives it is possible to generate permutations in a much more granular way: just on a single function or even on a group of lines. Only the code included in the foreach[q]
loop will be permuted (repeated number of permutations times), all the other code will be included only once. This means shorter compilation and execution / simulation times. The weak side of using m4 macros is the code size: since all permutations are included in the same binary executable file, the maximum available amount of data and program memory can be easily exceeded.
Taking into account the limitations of each approach:
It is also possible to use the mixed approach.
Wiki: Home
Wiki: Test
Wiki: z80
The last test template example for generate-cases.py does not work for me using python 3.8.5 on Mint 20/Ubuntu 20.04 with the sources for the 4.0.0 release archive. There are three problems:
Ending the comment in the
class:
line confuses the parser. The*/
should be in an empty line on its own:The template variable for the test case name should be
{type}
, not{types}
.Dynamically build up test function names are not recognized by the parser. A workaround would be to use an intermediate define:
How can I reproduce the issue?
I just tried running regression tests with PYTHON=python3.9 added, but don't see any problems.
P.S.: I didn't notice the problem report here in the wiki for quite a while. A bug report or support ticket would have been noticed earlier.