[LTP] RFC : refactor LTP execution model

[Garrett C. <yan...@gm...>]

Hi all,

   Currently LTP's execution scripts are split into 5 discrete blocks:

   1. execltp - python wrapper which invokes runltp and processes output to
expedite execution and reporting (provides summary and context for test
PASSes, and all non-PASSes).
   2. runalltests.sh - a semi-dynamic user-customizable execution wrapper
that executes runltp after performing needed setup for each section;
performs needed teardown after each section. Uses a number of scripts inside
testscripts/*.
   3. runltp - the primary end-user execution wrapper on top of ltp-pan.
   4. runltplite.sh - an abbreviated form of runltp (minus the email / html
output driver,
   5. testscripts/* - a number of ad hoc ltp-pan callers.

   If you take a look at the caller hierarchy, it's something similar to the
following:

| execltp |                     | runalltests.sh |
     |                                  |
| runltp  |  | runltplite.sh |  | testscripts/*  |
      \             |                  /
                | pan |

   Now, analyzing the layers in more detail:

   1. runltplite.sh contains the guts of runltp, but isn't maintained as
readily as runltp, thus needed bug fixes or feature enhancements aren't
making it into runltplite.sh as quickly, if at all, and as such there's some
unnecessary code divergence.
   2.  testscripts/* contains a lot of setup and teardown logic that could
be sorted out and properly handled inline before and after the test is
executed, unlike what's available currently.
   3. It would be nice if the commands and blocks of code were modularized,
s.t. outside groups could pre-process and/or post-process the tests to

   My proposal is as follows:

   1. A discrete set of reporting functions should be created (error, info,
warning, etc). This messages should adhere to the following basic paradigm:
       error - always displayed.
       info - only displayed when quiet isn't used.
       warning - only displayed when quiet isn't used.
       That way folks could
   2. runltp be split into discrete functional blocks, as follows:
       i.   Parse core options. Parse any additional options for
functionality like email, HTML output, etc.
       ii.  Prepare the tests based on the runtests files and options passed
in (run valgrind on the tests, etc).
       iii. Basic setup.
       iv.  Execute the tests.
       v.   Basic teardown.
       vi.  Output methods for email, HTML, etc.
   2. The testscripts/* setup and teardown logic be moved to a separate
script, which looks like the following:

<testname>_setup()
{
# Do setup here.
}

<testname>_test()
{
# Execute runtest file via pan.
}

<testname>_teardown()
{
# Do teardown here.
}

    So when runltp is executed, it does something like the following:
    1. source testscript [if it exists].
    2. Change the test ``execution plan'' so that they setup and teardown
inexplicably executed in the following manner (following Java's unit-test
and python unittest):

    Setup the test; if successful, continue on with the test(s) and teardown
at the end.

    The purpose for making this into a script as opposed to any other sort
of metadata, is that anyone executing the tests standalone will only need to
source the script to set everything up, then execute the test itself.
    The only thing that might be a problem is that runtest files are mashed
together if multiple runtest scenario files are specified. This shouldn't be
an issue though if everything is strung together properly via the calls
above (setup, test, teardown).
    Thoughts?
Thanks,
-Garrett