Hi all,

Thought I would provide some detail on an fwknop development effort around test coverage.  The fwknop test suite (test/test-fwknop.pl) has done a pretty good job of validating functionality for a while now, but a lot more tests are being added to try and get to 100% test coverage on a function, line, and branch basis.  It turns out that achieving "100%" coverage for all three of these is extremely hard, and that is a general statement for all software - not just fwknop.

Cutting to the chase, here are the current test coverage results as measure by the "gcov" tool with these .html views generated by "lcov":


So, in terms of functions, the test suite does fairly well with slightly over 99% of all functions covered.  Line coverage is less at about 85% and branch coverage is even harder at about 75%.  While this may not sound very good, it turns out that the usage of defensive checks (good for security) inflate these numbers.  That is, fwknop is careful to validate all sorts of things at various stages, and sometimes these checks are redundant making later checks harder to trigger than earlier ones. Another example is that all functions that deal with heap allocated memory have NULL checks as one would expect, but in practice it is hard to trigger a NULL result unless there is extreme memory pressure and this would affect test suite operations at the same time (and worse).  In some cases, it would be possible to account for these problems by building in special test coverage support into the fwknop code base, and some of this has already started.  For example, the new python fuzzer (see test/spa_fuzzer.py) leverages special #define's to assist with test coverage.  This approach could be extended to do things like maintain a global counter that is incremented with any malloc() (and friends) calls, and then force a NULL result whenever this counter hits any of a set of values during normal fwknop operations.

All of the above is run time testing, and is complementary to using static analyzers like Coverity.  The goal is to maintain an extremely high standard for the fwknop code base.