[Cunit-development] Need for deeper test hierarchy in CUnit

[Christophe M. <chr...@li...>]

Hi,
My name is Christophe Milard. I am working for ODP, OpenDataPlane, for
Linaro LNG, where we use C Unit for testing.
We are facing an issue where we would need a deeper hierarchy in the test
structure:
Currently, as I understand, the test registry can contain N suites, each
containing a certain number of tests. Each suite can also include a
initialisation and terminaison function.
ODP has a deeper hierarchy and some parts within this hierarchy, are
optional. We would like to be able to reflect this in the test environment.

To explain the situation, let me compare the situation with something more
illustrative like testing shell commands, like bash or busy box on
different platforms. Not all platforms supporting all commands.

In this case the wished structure may look as follows:

(R) is for Registrary, (S) is for Suite, and (T) is for test:

/shell(R)                            (the top entry, i.e. the registrary)
/bash/internal(S)                    (first hierarchy level: testing the
shell internal commands such as if, while, test)
/bash/internal/if(T)                 (test testing the 'if' command)
/bash/internal/while(T)              (test testing the 'while' command)
/bash/internal/test(T)               (test testing the 'test' command)
/bash/internal/...(T)                (test other internal commands)
/bash/fs(S)                          (first hierarchy level: testing the
shell commands related to file systems)
/bash/fs/ext2(S)                     (second level of hierarchy: testing
commands relative to the ext2 filesystem)
/bash/fs/ext2/mount(T)               (testing command 'mount' for ext2)
/bash/fs/ext2/umount(T)              (testing command 'umount' for ext2)
/bash/fs/ext2/...(T)                 (testing other ext2 commands)
/bash/fs/fat(S)                      (second level of hierarchy: testing
commands relative to the fat filesystem)
/bash/fs/fat/mount(T)                (testing command 'mount' for fat)
/bash/fs/fat/umount(T)               (testing command 'umount' for fat)
/bash/fs/network(S)                  (second level of hierarchy: testing
commands relative to networked filesystem)
/bash/fs/network/nfs(S)              (third level of hierarchy: testing
commands relative to nfs networked fs)
/bash/fs/network/nfs/mount(T)        (testing command 'mount' for networked
file system nfs)
/bash/fs/network/nfs/umount(T)       (testing command 'umount' for
networked file system nfs)
/bash/fs/network/samba(S)            (third level of hierarchy: testing
commands relative to windows networked fs)
/bash/fs/network/samba/mount(T)      (testing command 'mount' for networked
file system samba)
/bash/fs/network/samba/umount(T)     (testing command 'umount' for
networked file system samba)

The initialization function of /bash/fs could create a valid device (e.g.
/dev/hda) and the initialization function of /bash/fs/ext2 would populate
/dev/hda with a valid ext2 filesystem.

Some systems may only implement part of the API. A system could, for
instance, include internal commands, ext2 and nfs filesystems, but not fat
nor samba. On such a system, one would be willing to select and run suites
/bash/internal, /bash/fs/ext2, and /bash/fs/network/nfs, but none of the
others.

This example hopefully shows the need we have.

I can see two possible approaches to improve CUnit to match this need:

**The first one (cheap :-) is to add some functions in the API:*
mainly:
CU_get_suite_by_name_pattern(const char* pattern, CU_pTestRegistry
pRegistry, int occurrence)
CU_get_test_by_pattern(const char* pattern, CU_pSuite pSuite, int
occurrence)

These two functions would search the suite or test by name matching the
pattern and return the occurrence-th (+1) match.
Pattern could either be a full name, or a name finishing with '*'.
For instance:
CU_get_suite_by_name_pattern("suite1", R, 0) would behave as
CU_get_suite_by_name("suite1",
R).
CU_get_suite_by_name_pattern("suite2*",R, 3) would return the 4th suite
whose name start by "suite2" (or NULL if none).

Also the interactive mode might need some modifications to handle pattern
matching.

This alternative provide a way to achieve some kind of hierarchy simply by
having the test or suite names matching their hierarchical position (e.g
"/bash/fs/network/samba/mount"). Running all occurrences of tests matching
pattern "/bash/fs/network*" would run all tests testing networked
filesystems.
But it does not provide the ability to have one initialization function per
hierarchical level.

**The second one (more complex) would be to make CUnit trully recursive.*
in other words to let a test suite contain another one. This would
certainly have a much larger impact on CUnit and the current API.


Note that both solution should be able to coexist, as far as I can see it.
So going for the first solution should not prevent the second (better?)
solution to be implemented later.

This first mail is mainly meant to open a discussion: is there already
something to handle hierarchy that I missed? If not, do anyone see a better
solution than the two depicted above?... if not, what is the best way to
contribute to improve CUnit? :-)

Thanks in advance for your feedback,

Christophe Milard