Re: [LTP] [RFC] [OLS 2008 FALLOUT] Issue # 1

[Lucas M. R. <lu...@br...>]

On Wed, 2008-08-13 at 15:16 -0600, ro...@us... wrote:
> To add to this discussion, I think the folks in the Autotest community
> would be more likely to be involved in this issue than the LTP. You
> may want to look at what the LTC team in Brazil has been doing with
> the Autotest community around test automation.
> 
> -Robbie Williamson

I've briefly looked over the thread, and one thing I'd like to point out
is: we must focus the projects on what they do better. LTP is a large
and well established linux kernel *test suite*. So IMHO what we can do
to improve LTP should be focused on making it a better linux kernel test
suite. 

 * Improving test API, 
 * Improving validity and correctness of the test cases, 
 * Improving log generation system, 
 * Improving results summary

And stuff on the realm of a test suite. Yes, installing distros is a
common use case, building kernels is a common use case, the requirements
come from the community, but guess what... LTP is not the right place to
put this functionality, and there are already projects that do this.
I've exposed my thoughts about the current state of open source test in
general, and linux distros testing in general here:

http://mybravenewworld.wordpress.com/2008/08/06/a-tale-about-open-source-testing-part-1-autotest/

We as a quality control team for linux distros are spending quite a lot
of time to port and adequate our distro testing to the autotest
framework, and it's been really worth the effort spent so far. LTP is
one of the pieces that makes up for *great* quality control on linux
testing, but trying to make it the 'one stop shop' for the whole linux
distro testing activity is not a wise use of our resources.

So let's focus our efforts in a rational way, let's integrate LTP as a
great test suite that it is with the open source projects that can do
kernel build and distro deploy, work on those projects, and live happily
and in peacefully :)

Please take my comments as a grain of salt.

> Inactive hide details for pl...@li...pl...@li...
> 
> 
>                                 pl...@li... 
>                                 
>                                 08/13/2008 03:36 PM
>                                 
> 
>                To
> 
> Randy Dunlap
> <rd...@xe...>
> 
>                cc
> 
> su...@li..., ltp-list <ltp...@li...>, Robert Williamson/Austin/IBM@IBMUS, Paul Larson/Austin/IBM@IBMUS, Michael Reed/Austin/IBM@IBMUS, Nate Straz <na...@re...>, martinjn <mar...@us...>, crackerjack-devel <cra...@li...>, Jeff Burke <jb...@re...>, David Howells <dho...@re...>, Masatake YAMATO <ya...@re...>, Cai Qian <qc...@re...>, Patrick Kirsch <pk...@su...>, Peter 1 Oberparleiter <Pet...@de...>, Mike Frysinger <va...@ge...>, Renaud Lottiaux <Ren...@ke...>, Garrett Cooper <yan...@gm...>, Michael Kerrisk <mtk...@go...>, Vijay Kumar <vij...@br...>, Christian Di Biagio <cdi...@gm...>, Serge Hallyn <serue@lin>
> 
>           Subject
> 
> Re: [RFC] [OLS
> 2008 FALLOUT]
> Issue # 1
> 
> 
> 
> Agreeing with what pretty much everyone else has said, I'd also like
> to
> add that this isn't the first time someone has requested something
> like
> this. Iirc, there was a small attempt at automating at least some part
> of it, you may still find artifacts of it lingering around in the
> scripts directory.  Other alternatives include things like autotest,
> STP
> at OSDL, or even rolling your own.  As Randy suggested, it isn't all
> that hard to automate kernel build and boot.  But end to end
> automation
> normally requires some level of integration with your hardware/network
> infrastructure.  It can be done in a generic enough way, sure, but
> normally ends up spiraling out of control into something difficult to
> setup and manage.  People tend to look at something like that and
> decide
> they could come up with something simpler (but not generic enough to
> be
> used anywhere) themselves.
> 
> -Paul Larson
> 
> On Wed, 2008-08-13 at 09:43 -0700, Randy Dunlap wrote:
> > On Wed, 13 Aug 2008 14:02:55 +0530 Subrata Modak wrote:
> > 
> > > Hi,
> > > 
> > > Recent OLS 2008 was a critical point in LTP´s evolution, as i got
> the
> > > opportunity to meet several people across the Linux ecosystem, and
> > > listened to their opinion about LTP. Here i would start a mail
> chain
> > > with the above Subject line, discuss each and every issue in this
> > > mailing list, collate everybody´s opinion on those issue(s) and
> take
> > > action accordingly. These are the people i encountered:
> > > 
> > > 1) People, who uses LTP heavily. And they suggested lots of
> improvement
> > > to it. We will discuss those issues in mails from now,
> > > 
> > > 2) People, who have heard about LTP and not used it till now. They
> > > promised that they will give a try,
> > > 
> > > 3) People, who has never heard about it. So, it was an opportunity
> to
> > > convey them what LTP is all about. I hope people in Category 2 & 3
> will
> > > start using LTP soon, and we will get an enlarged user base and
> hence
> > > bringing more contribution in future.
> > > 
> > > =================
> > > ISSUE # 1
> > > =================
> > > The heavy users made a point of LTP having the capability to
> automate
> > > testing completely. What they meant was LTP to have capability to
> do:
> > > 1) Kernel Build,
> > 
> > With what/how many configs?
> > 
> > > 2) Kernel Install/Distro install,
> > 
> > With which/how many distros?  This is an unending mess...
> > and $someone will say "you don't support my $distro".
> > 
> > 
> > > 3) Then do specific/all tests,
> > 
> > (Like Serge said) Just focus here.
> > 
> > 
> > > They said that this feature will simplify the way they work. I
> would
> > > like to know what you all think about this.
> > > 
> > > What i feel is, every project should evolve and should be flexible
> > > enough to meet their users requirement dynamically, and should not
> be
> > > tied down with the limitations of it´s initial design constraints.
> If
> > > automating kernel build, install and tests is a requirement coming
> from
> > > the user community, then we need to give a hard look at it. I
> would like
> > > to know what you think about this.
> > 
> > Kernel builds are relatively easy to do & automating them isn't
> difficult.
> > OTOH, installing a distro is a much larger task in my experience.
> > 
> > 
> > ---
> > ~Randy
> > Linux Plumbers Conference, 17-19 September 2008, Portland, Oregon
> USA
> > http://linuxplumbersconf.org/
> 
> 
> 
> 
-- 
Lucas Meneghel Rodrigues

IBM STG, Linux Technology Center
Office:   +55 19 2132-3673
Mobile:   +55 19 9199-9487
Home:     +55 19 2121-9487
Tie Line: 839-3673
E-Mail/Sametime: lu...@br...