Re: [Stp-devel] Re: [LTP] Re: [nptl] Demonstration available

[Gerrit H. <gh...@us...>]

On Thu, 13 Jan 2005 17:25:37 PST, Bryce Harrington wrote:
> On Thu, 13 Jan 2005, Gerrit Huizenga wrote:
> > The key of this tool is to reduce the amount of work to interpret
> > the ouptut.  We are in a good state across the board of being able
> > to get lots of output and lots of results; I'm now more concerned about
> > recognizing trends, e.g. John Cherry's work to consolidate warning
> > counts, etc., would be good to have the same thing from LTP and other
> > test harnesses.
> >
> > Pretty soon, I'd love to see a dashboard with a blinking red light
> > whenever a new bug went into linux that can be clicked, drill down
> > to the failure, click through to the changed lines of code, and
> > work on a fix.
> >
> > Dreaming of a brighter, not too distant future.  ;-)
> 
> Have you seen this LTP page I did a while back?
> 
> http://developer.osdl.org/bryce/ltp/
 
Nope - I hadn't seen that.  But the trend lines don't look good,
which probably indicates it wasn't widely published.  ;-)  In theory,
if that kind of data gets published with trend lines, there is a
tendancy for people to fix things, be it kernel-janitors, test team
folks looking at the constant failures, etc.

> It's not 100% automated, but I do have some scripts to generate that
> data that do most of the work.  It doesn't give the specific line number
> but does give the path to the source code for the failed test, so that
> narrows you in pretty close.

One of the things that our test team was working on was building an
inventory of gcov output as relates to individual tests.  Their goal
has been to identify automatically which tests to run when a given
line of code changes.  However, it seems like this could be used in
reverse as well, possibly to use such a directory to tell which lines
of code might have been changed when a section of code fails.  I'm
wondering if this is something that Nigel might be interested in
looking into with the LTP team as that falls into his code coverage
area/research work quite nicely.

> This was useful for spotting the bug that occurred around
> patch-2.6.7-bk11.  This gave us the specific day to check for the patch
> that caused the problem, and were able to zero in on the specific patch
> pretty quickly.
> 
> Would you be interested in working together to make this more capable?

What in particular do you think will need to be done?  And would it
tie in at all with the work that Sebastien has done?  I'm also interested
in solutions that are sufficiently portable that we could use them
both at OSDL but also within our firewall in our automated test
harness.  So, if this is a packaged set of code that his portable,
we might be able to find someone to work help with this.

gerrit