[Kerncomp-devel] Re: [ANNOUNCE] Automated Kernel Build Regression Testing

[Darren W. <ds...@ge...>]

Hi Ian

On Wed, 11 May 2005, Ian Wienand wrote:

> On Wed, May 11, 2005 at 02:32:39PM +0200, Jan Dittmer wrote:
> > But as I can see from your output you also build if nothing has
> > changed in the git tree (as is the case since -rc4).
>=20
> Ahh, that might be a bug :) We didn't used to build when nothing
> changed with BK.  Darren?

We have always built nightly, it would be trivial to set up kerncomp
to only build when things change.

I need to check in a few changes before Friday PM, I'll add it to the TODO
list.
=20
>=20
> > By grabbing the patches from kernel.org, I've definite checkpoints
> > what was tested and you can fully rebuilt the build environment
> > locally. For grabbing the different trees I use a tool called
> > 'ketchup'.
>=20
> Good point.  BK used to make it really easy to update your tree with
> what was pulled from the logs; we will see how this works out with
> Cogito.

Trivial once again since any applied changes updates the HEAD which
can be written to the logs. It has never really been an issue for me
since I start fixing broken builds at or near the date of the break
with the up to the minute GIT/BK tree, so if a fix has been committed
to mainline I get that change and stop fixing, get on with life and
tommorrow build is all OK.

Add to TODO since I see problems here.

>=20
> > Also the -mm trees have much more experimental stuff in them and
> > are more interesting to track.
>=20
> Yes; these should definitely be tracked.  There have been examples
> where IA64 was broken in -mm for a long time because noone ever tries
> it.
>=20

For IA64 centric stuff I think the ia64 testing tree would also be relevant,
this tree need testing before these changes hit any mainline or -mm tree.
This is the advantage of GIT, we can point at any branch we want and pull
=66rom it and GIT will manage the merging (in theory). For instance I cloned
a kernel.org Git tree for local use here which is periodically updated
every 1/2 hour. The kerncomp GIT tree use to pull from the same kernel.org
tree, with a single command (cg-add-branch <branch-name> <URI>)and a small
tweak to the kerncomp script (update from branch-name), kerncomp now pulls
=66rom our local tree, this is a trivial example. =20
=20
> > ls arch/*/configs/* gives about 184 different defconfigs. A test run
> > on my single workstation would take literally ages. Therefore I'd
> > consider a distributed client/server approach.

And not all cross-compilers successfully build the kernel, where as the
native build would succeed. There have been discussion about this on lkml.=
=20

>=20
> For a broad approach like your cross compilers; for sure.  For other
> architectures testing those defconfigs can give you pretty much
> complete coverage. =20
>=20
> > That's a bit more generalized.  I think it's focussing more on
> > runtime than compile testing.
>=20
> I'd love to have better runtime testing; for example running LTP on
> the booted kernel in our simulator.  It's one thing for the kernel to
> build, another thing for it to work :)
>=20
> > I though about doing runtime testing with qemu or by using a network
> > of different arch machines. But at home I only have i386 and sparc32
> > so I considered it rather pointless. And qemu is also not that stable
> > that I'd trust the results.

In any simulator, LTP or any other thorough testing would take years, we ne=
ed
real hardware, this is my next big project (Autobench/test). This I think
would be really cool. I think we could get the resources, we just need the
auto stuff. Looking behind me I see idle sparc, ppc, alpha, 386 .....

>=20
> Luckily IA64 has the ski simulator which, on the whole, is really
> good.  You are right; anything is possible with a bunch of machines!
>=20
> > Same are mine. And currently very focussed on kernel building. I'd have
> > to look over them, and then I could post them if you're interested.
>=20
> I would be happy to give you a CVS tree in our SF project if you are
> thinking of releasing your code.  That way at least resources are
> together.
>=20
> > But I looked at yours, and you're not using a db at all? You're just
> > using flat files to store the results and parse them afterwards?
>=20
> Yes ... I think this has many advantages such as compressing and
> archiving the logs, and the fact you can simply rsync things around.
> I'm not married to it though ...

And you do not require a dbm, hey we could store the results in a
Git tree (8-0

>=20
> -i

 - dsw

--------------------------------------------------
Darren Williams <dsw AT gelato.unsw.edu.au>
Gelato@UNSW <www.gelato.unsw.edu.au>
--------------------------------------------------