[Crackerjack-users 00066] [LTP] [RFC] [OLS 2008 FALLOUT] Issue # 2

[Subrata M. <su...@li...>]

Hi,

Thank you very much for the feedback/opinion that you provided regarding
Issue #1 for kernel-Autobuild in LTP. I will study everybodys´ comments
and reply back soon.

However, here goes Issue #2 (In-fact contains more sub issues). It comes
from somebody who works very closely with The Linux Foundation. It seems
the following issues in LTP should be addressed most urgently:

2.1) All LTP tests should run faster,

2.2) LTP should give clear results, either PASS or FAIL. People are not
interested in Partial results, i,e, they should not bother about BROK,
CONF, WARN, RETR, etc stuff. What they want is either PASS or FAIL. The
basic message is that kernel developers may not become experts in LTP,
but they should be able to decipher the results easily by pointing to
exactly PASS-ed or FAIL-ed,

2.3) Cleanup tests that hardly matter any more as the kernel has evolved
quite stable, like syscalls tests for read(), write(), etc for which the
possibility for introduction of regressions issue(s) are very less. They
want these tests to be removed from LTP.

While i personally do not favour removing any tests from LTP, what can
be done is to disable some commonly agreed tests, which will not run by
default. If you think, you have such lists, then i can disable them from
running, and, not completely removing them from LTP,

2.4) Fix LTP´s false positives.

There has been such complaints in the past that certain tests throw up
messages they should not have done under that circumstance(s). But i
have not exactly understood the concept of false positives. When i tried
to create them, i was not able to. I have tried to address this issue in
my paper,

2.5) LTP should start focusing on IN-KERNEL-API testing, basically
testing the API´s which are available only inside the kernel.

This is something which i liked very much. Till now we have been doing
testing only from the user-space. But if we can do testing from within
the kernel- space, then we can increase our testing effort(s) for the
device drivers as well. Having said that, if you agree to this proposal,
we/somebody needs to come up with a template of how such a test should
be written and Publish that in:
http://ltp.sourceforge.net/documentation/how-to/ltp.php,
so that it becomes a LTP standard. But we will have some challenges of
how the test case(s) will report results/pass/fail back to the
log/output file(s), as i believe, such kind of testing has to be done
through loading and unloading of Kernel Modules. Probably (you can
propose a better one) we will follow something like:

1) Script that loads/unloads the module (in-kernel test case) - reports
any error in loading/unloading. We will handle this through our tst_*
set of functions,
2) The actual module has to test the API in the kernel-space and somehow
report the result using, say, the dmesg interface,
3) The script should be able to parse out that message and report with
the tst_* set of functions,

I would again like to know your thoughts on this.

Regards--
Subrata