Re: [LTP] [RFC] [OLS 2008 FALLOUT] Issue # 2

[Nate S. <nat...@re...>]

On Aug 18 19:45, Subrata Modak wrote:
> 2.1) All LTP tests should run faster,

IIRC, most of the LTP tests run very fast.  Most can be run in parallel
with other tests without conflicting.  Which tests are getting
complaints?

> 2.2) LTP should give clear results, either PASS or FAIL. People are not
> interested in Partial results, i,e, they should not bother about BROK,
> CONF, WARN, RETR, etc stuff. What they want is either PASS or FAIL. The
> basic message is that kernel developers may not become experts in LTP,
> but they should be able to decipher the results easily by pointing to
> exactly PASS-ed or FAIL-ed,

You can thank those same developers for making it not that simple.
Perhaps those developers would like to map BROK, CONF, WARN, RETR to
PASS or FAIL.  From there we can disprove their assertion.

> 2.3) Cleanup tests that hardly matter any more as the kernel has evolved
> quite stable, like syscalls tests for read(), write(), etc for which the
> possibility for introduction of regressions issue(s) are very less. They
> want these tests to be removed from LTP.
> 
> While i personally do not favour removing any tests from LTP, what can
> be done is to disable some commonly agreed tests, which will not run by
> default. If you think, you have such lists, then i can disable them from
> running, and, not completely removing them from LTP,

It all comes down to risk management.  Perhaps we can use code coverage
to disable tests which are completely covered implicitly by other tests.

> 2.4) Fix LTP´s false positives.

If a test is passing when it should be failing, then we need to add more
code to the verification steps to make it catch the failure.

> 2.5) LTP should start focusing on IN-KERNEL-API testing, basically
> testing the API´s which are available only inside the kernel.

This would be great to see.  You'll have to get into the design
discussions early so the code is written in a testable way.  Adding a
unit test framework in the kernel would be interesting.

Nate