[Kerncomp-devel] Re: [ANNOUNCE] Automated Kernel Build Regression Testing

[Jan D. <jdi...@pp...>]

Ian Wienand wrote:
> On Wed, May 11, 2005 at 09:23:59AM +0200, Jan Dittmer wrote:
> 
>>I hope you know of my http://l4x.org/k effort? It features the possibility
>>to cross-compile all archs, built different configs and has a somewhat
>>nice web interface.
> 
> 
> That's a great interface!  I like the log on mouseover bit.  An RSS
> feed is something that I've found extremely useful; I can check the
> results when I get in of a morning along with all my other RSS feeds.

It would be trivial to add as it is database driven. Perhaps I find
some time in the evening...

>>It does fully automatic defconfig builds on 23 platforms for -git,
>>-rc and -mm releases.
> 
> 
> Ok, so you're grabbing the patches from kernel.org?  We have found it
> useful to do the pull directly via git (well, used to be BitKeeper)
> every night to keep right up to date.

But as I can see from your output you also build if nothing has
changed in the git tree (as is the case since -rc4).
By grabbing the patches from kernel.org, I've definite
checkpoints what was tested and you can fully rebuilt the build
environment locally. For grabbing the different trees I use
a tool called 'ketchup'.
Also the -mm trees have much more experimental stuff in them and
are more interesting to track.

> We also find it useful to build all the defconfigs for IA64; some
> other architectures also have multiple defconfigs for different
> classes of machine.  Luckily, for IA64 the defconfig builds probably
> cover > 95% of machines out there (since there's only a few vendors)
> ... the x86 situation is obviously different!

ls arch/*/configs/* gives about 184 different defconfigs. A test run
on my single workstation would take literally ages. Therefore I'd
consider a distributed client/server approach.

>>What I would really like to see is a client/server architecture where
>>multiple machines (over the inet?) can share the build results, so that
>>one can start offering patch testing, ie. someone uploads a patch against
>>some version and the network test the resulting source against all archs
>>and all requested configs. That could be really handy I can imagine.
>>A full defconfig run on all 23 archs currently takes about 2 hours on my
>>2.8ghz xeons.
> 
> 
> Yes, that would be very nice; I think OSDL provide something similar
> to that (http://www.osdl.org/lab_activities/kernel_testing/stp/) but
> I've found their interfaces to be a little difficult at times.

That's a bit more generalized.  I think it's focussing more on runtime
than compile testing.
Making such a thing work with non-Kernel related stuff would of course
be nice too.
I though about doing runtime testing with qemu or by using a network
of different arch machines. But at home I only have i386 and sparc32
so I considered it rather pointless. And qemu is also not that stable
that I'd trust the results.

>>I'll check out your sourcebase soon. My scripts are about 400 lines for
>>the builder and 1100 lines for the web interface, all php using
>>postgres as a backend.
> 
> 
> Ours is nothing fantastic, just hacked together shell and PHP code,
> but it works.

Same are mine. And currently very focussed on kernel building. I'd have
to look over them, and then I could post them if you're interested.
But I looked at yours, and you're not using a db at all? You're just
using flat files to store the results and parse them afterwards?

Regards,

-- 
Jan