Bryce, I looked at your script.  It is the sort of thing I have in mind, so adapting it is a good way to proceed.  Specifically, here is the information and process I think could be helpful to LTP users --  **LTP developers and tester, please let me know what you think **.

The process could be incorporated into 'ltprun' or it could take ltprun results as input and proceed as follows:

- capture test configuration (metadata)
- query archive using config and current results
- return exceptional results: tests that fail;
tests that typically fail but passed this time, etc.

- report to the user:
        - config: relevant test/kernel/HW config info
        - historical summary: failure statistics
        - bugs: associated bug reports numbers
        - note: log of comments
        - other options:
                - official LTP explanation
                - test description


One of the things to resolve is determining how broad the query should be. For example, how specific do we get about hardware configuration and kernel options?  And obviously we need to adapt the existing mechanism for developers/testers to enter new results and notes.


Here is a fake example of the new output that would follow the usual LTP report.


===================================================================
Test name: nanosleep02 [may also include version id]

config: x86

Historical Statistics: failed 45% of tests run in the last 6 months

bugzilla.ibm.com: 93450, 32455

Notes:

Sep 15 2004 (Nigel Hinds)

This failure is due to a hardware clock resolution problem.


===================================================================



Bryce Harrington <bryce@osdl.org>
Sent by: ltp-list-admin@lists.sourceforge.net

01/13/2005 08:14 PM

       
        To:        Nigel Hinds/Watson/IBM@IBMUS
        cc:        gh@us.ltcfwd.linux.ibm.com, <ltp-list@lists.sourceforge.net>, <nptl@bullopensource.org>, <stp-devel@lists.sourceforge.net>, TSLogParser <tslogparser-discuss@lists.sourceforge.net>
        Subject:        Re: [LTP] Re: [nptl] Demonstration available



You may want to look at a script I wrote that is available in ltp/tools/
called STPfailure_report.pl.  This retrieves results from a results
server (such as STP) and generates additional data about the failures by
extracting the comments section from the failed test(s).

I have found it quite useful to use this script to pull results from two
test runs, and then just compare them using diff.

This was designed to run in conjunction with the LTP reports generated
in STP, but is generic enough that it can probably be adapted to other
LTP harnesses.

Bryce

On Thu, 13 Jan 2005, Nigel Hinds wrote:
> Gerrit & Bryce,
>
> My intent is to maintain results and provide an LTP option that would
> query the old data and return relevant previous results with the new
> results.  The idea is that this would provide some context in which an LTP
> user can evaluate the current results.  It would be great to make use of
> any existing systems and/or data.  I suspect the prototype of any new
> service would be hosted at IBM LTC in Austin. But don?t quote me on that.
>
>
>
>
>
>
> Bryce Harrington <bryce@osdl.org>
> Sent by: <bryce@osdl.org>
> 01/13/2005 01:14 AM
>
>         To:     gh@us.ltcfwd.linux.ibm.com
>         cc:     Nigel Hinds/Watson/IBM@IBMUS,
> <ltp-list@lists.sourceforge.net>, <ltp-list-admin@lists.sourceforge.net>,
> <nptl@bullopensource.org>, <stp-devel@lists.sourceforge.net>, TSLogParser
> <tslogparser-discuss@lists.sourceforge.net>
>         Subject:        Re: [LTP] Re: [nptl] Demonstration available
>
>
> OSDL has been hosting some tools for doing automated LTP runs, and you'd
> be quite welcome to build on what we've got.
>
> We've got triggers that watch for and grab major and minor kernel
> releases and run LTP on it.  Areas where more work is needed includes
> hooking up some of the non-default parts of LTP, and something for doing
> results comparisons / graphing.
>
> Bryce
>
> On Wed, 12 Jan 2005, Gerrit Huizenga wrote:
> >
> > Excellent Nigel - any idea where this might get hosted?
> >
> > gerrit
> >
> > On Wed, 12 Jan 2005 22:43:39 EST, Nigel Hinds wrote:
> > >
> > > Gerrit, I?m trying to get an automated results capture project started
> for
> > > LTP.
> > >
> > > nigel
> > >
> > > gh@us.ltcfwd.linux.ibm.com
> > > Sent by: ltp-list-admin@lists.sourceforge.net
> > > 01/12/2005 03:15 PM
> > > Please respond to gh
> > > =20
> > >         To:     nptl@bullopensource.org
> > >         cc:     TSLogParser
> <tslogparser-discuss@lists.sourceforge.net>,=20
> > > stp-devel@lists.sourceforge.net, ltp-list@lists.sourceforge.net
> > >         Subject:        [LTP] Re: [nptl] Demonstration available
> > >
> > >
> > >
> > > On Wed, 12 Jan 2005 19:04:28 +0100, Sebastien Decugis wrote:
> > > > Hi,
> > > >
> > > > You can see the new release (v03) of the TSLogParser in action at:
> > > > http://tslogparser.sourceforge.net/run-browse.php
> > > >
> > > > You can access the administration interface there, and see how the
> tool
> > > > can be integrated in a website (the Look & Feel, menus, ...)
> > > >
> > > > Note: The database is reset twice an hour in case someone deletes
> the
> > > > content.
> > > >
> > > > Best regards,
> > > > Seb.
> > >
> > > Sebastien - this is *really* cool.  It would be neat if STP and
> > > or regular LTP runs could be put up on a web site which can be
> > > browsed for historical data.  Anyone from LTP or STP planning to
> > > do something like that?
> > >
> > > Oh, the only part that *isn't* as cool is the number of failures
> > > reported.  ;-)
> > >
> > > gerrit
> > >
> > >
> > > -------------------------------------------------------
> > > The SF.Net email is sponsored by: Beat the post-holiday blues
> > > Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
> > > It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt
> >
> >
> > -------------------------------------------------------
> > The SF.Net email is sponsored by: Beat the post-holiday blues
> > Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
> > It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt
> > _______________________________________________
> > Ltp-list mailing list
> > Ltp-list@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/ltp-list
> >
>
>
>



-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list