[Kerncomp-devel] Re: [ANNOUNCE] Automated Kernel Build Regression Testing

[Ian W. <ia...@ge...>]

On Wed, May 11, 2005 at 02:32:39PM +0200, Jan Dittmer wrote:
> But as I can see from your output you also build if nothing has
> changed in the git tree (as is the case since -rc4).

Ahh, that might be a bug :) We didn't used to build when nothing
changed with BK.  Darren?

> By grabbing the patches from kernel.org, I've definite checkpoints
> what was tested and you can fully rebuilt the build environment
> locally. For grabbing the different trees I use a tool called
> 'ketchup'.

Good point.  BK used to make it really easy to update your tree with
what was pulled from the logs; we will see how this works out with
Cogito.

> Also the -mm trees have much more experimental stuff in them and
> are more interesting to track.

Yes; these should definitely be tracked.  There have been examples
where IA64 was broken in -mm for a long time because noone ever tries
it.

> ls arch/*/configs/* gives about 184 different defconfigs. A test run
> on my single workstation would take literally ages. Therefore I'd
> consider a distributed client/server approach.

For a broad approach like your cross compilers; for sure.  For other
architectures testing those defconfigs can give you pretty much
complete coverage.  

> That's a bit more generalized.  I think it's focussing more on
> runtime than compile testing.

I'd love to have better runtime testing; for example running LTP on
the booted kernel in our simulator.  It's one thing for the kernel to
build, another thing for it to work :)

> I though about doing runtime testing with qemu or by using a network
> of different arch machines. But at home I only have i386 and sparc32
> so I considered it rather pointless. And qemu is also not that stable
> that I'd trust the results.

Luckily IA64 has the ski simulator which, on the whole, is really
good.  You are right; anything is possible with a bunch of machines!

> Same are mine. And currently very focussed on kernel building. I'd have
> to look over them, and then I could post them if you're interested.

I would be happy to give you a CVS tree in our SF project if you are
thinking of releasing your code.  That way at least resources are
together.

> But I looked at yours, and you're not using a db at all? You're just
> using flat files to store the results and parse them afterwards?

Yes ... I think this has many advantages such as compressing and
archiving the logs, and the fact you can simply rsync things around.
I'm not married to it though ...

-i