Re: [Crackerjack-devel] [LTP] [RFC] [OLS 2008 FALLOUT] Issue # 1

[Louis R. <Lou...@ke...>]

On Wed, Aug 13, 2008 at 02:02:55PM +0530, Subrata Modak wrote:
> Hi,
> 
> Recent OLS 2008 was a critical point in LTP´s evolution, as i got the
> opportunity to meet several people across the Linux ecosystem, and
> listened to their opinion about LTP. Here i would start a mail chain
> with the above Subject line, discuss each and every issue in this
> mailing list, collate everybody´s opinion on those issue(s) and take
> action accordingly. These are the people i encountered:
> 
> 1) People, who uses LTP heavily. And they suggested lots of improvement
> to it. We will discuss those issues in mails from now,
> 
> 2) People, who have heard about LTP and not used it till now. They
> promised that they will give a try,
> 
> 3) People, who has never heard about it. So, it was an opportunity to
> convey them what LTP is all about. I hope people in Category 2 & 3 will
> start using LTP soon, and we will get an enlarged user base and hence
> bringing more contribution in future.
> 
> =================
> ISSUE # 1
> =================
> The heavy users made a point of LTP having the capability to automate
> testing completely. What they meant was LTP to have capability to do:
> 1) Kernel Build,
> 2) Kernel Install/Distro install,
> 3) Then do specific/all tests,
> 
> They said that this feature will simplify the way they work. I would
> like to know what you all think about this.
> 
> What i feel is, every project should evolve and should be flexible
> enough to meet their users requirement dynamically, and should not be
> tied down with the limitations of it´s initial design constraints. If
> automating kernel build, install and tests is a requirement coming from
> the user community, then we need to give a hard look at it. I would like
> to know what you think about this.

[ Sorry for arriving late in the discussion (hollydays) ]

I definitely agree with others that 1) and 2) should be done in separate
projects.

However, 3) (doing selected tests) has to be done in LTP, IMHO.

AFAICS, what LTP currently provides for this is (please correct me where I am
wrong):
- run all tests,
- run a (whole) suite of tests (eg "syscalls"),
-? run a single test.

What I would like:
i) run a custom subset of tests, out of existing suites (eg not all "syscalls"),
ii) disable some tests in a suite (eg disable fork07 in "syscalls"), but still
   report that the test was disabled.

For i), why not writing my own file under runtest/? Because some tests need a
few setup before being run, and I don't want to copy these setup in all custom
runlists.
	To address this, I suggest to move those setup commands in
dedicated executables located together with the test sources (say abort01.setup
under testcases/kernel/syscalls/abort/). Those *.setup could be simple shell
scripts (abort01.setup would contain "ulimit -c 1024") or binaries as well.

For ii) a usecase is the following: when implementing a new feature, often the
feature is buggy and breaks several tests for different reasons. "Break" means
the test result is "FAIL", or the machine was crashed as well. As long as
fixing those bugs is fast, one can simply comment out the failing tests in the
related file under runtest/. However, when it takes a long time to fix those
bugs, having a report giving the list of successful tests as well as the ones
that were (intentionally) disabled gives some bugfix progress bar.
	To address this, I suggest to, first, do as I suggested for i), and
second, replace the setup code in the runlists by wrapping code when desired
Example to disable fork07:
	fork07	fork07 
becomes
	fork07	disabled fork07

Where "disabled" is a script living in the same directory as test binaries, that
contains:

<begin script>
#!/bin/sh

echo $1 disabled
<end script>

If what I suggest looks ok, I can write  patches.

Thanks,

Louis

-- 
Dr Louis Rilling			Kerlabs
Skype: louis.rilling			Batiment Germanium
Phone: (+33|0) 6 80 89 08 23		80 avenue des Buttes de Coesmes
http://www.kerlabs.com/			35700 Rennes