[Stp-devel] LTP STP-Lite

[Bryce H. <br...@os...>]

I've been working this afternoon on figuring out how to do an "STP-Lite"
version of LTP, so went through some of the past emails about it, and
Mary's document from several months back, to try an work up a more
generalized doc on it.  Let me know what you think.

Bryce



Scalable Test Platform - Lite
-----------------------------

STP/Lite is a system of packaging Linux tests in a manner that allows
running the tests directly or within the STP framework proper.  STP/Lite
consists of documentation and scripts that systematizes the installation
and running of tests.  In doing so, it establishes a consistent
interface for running tests that will make running the tests easier by
individuals and straightforward to automate through a test harness
framework.


Background
==========
The Scalable Test Platform (STP) is a powerful application developed at
the Open Source Development Labs for performing testing of patches to
the Linux kernel.  In order to fit a wide range of tests into the
system, a common interfacing approach was established to control how
data was passed into the test and results organized and returned.  As
well, for a number of tests, some modifications were performed to
enhance their reporting capability, eliminate hangs/blocks, and so
forth.  As a consequence, users who try out the original test are not
always able to match up the output to the STP run, which makes local
comparison to STP runs difficult.

STP itself can be downloaded, installed, and run, but many users are
interested only in the specific test, and having to set up a database,
web server, etc. to erect the full STP framework is overkill.  They want
to simply download the test being run by STP, run it on their local
system, and be able to directly compare results.

Thus the motivation for STP-Lite is to provide packaged, easily
installable versions of the tests run in STP that include the same
results generation tools used in STP and produce directly comparable
results.  STP itself is not required for running STP-Lite enabled
tests.


Assumptions
===========
In the design for STP-Lite we make certain assumptions about the user
and their skillset, to help establish what the user will expect from the
package and what level of skill we can expect from the user.

We assume the user to have basic system administration skills, thorough
familiarity with using Linux at the commandline, and sufficient control
over the testing box that they can install software and reboot as
needed.  Specific skills assumed:

   * Linux commandline, setting environment variables, editing config
     files like /etc/sysctl.conf, etc.
   * System administration, compiling and installing software, etc.
   * Compiling and installing Linux kernels
   * Basic knowledge of workloads or applications used by the test
   * Moderate knowledge of the test itself
   * Basic scripting (shell, Perl, etc.) capabilities
   * Understands written English
   * Basic familiarity with STP from a user perspective
   * No knowledge of STP internals or design


STP-Lite Library
================
The general scripts and code used in STP-Lite-enabled test packages is
provided through a common library, called stp_lite.  This library
consists of the following components:

   * stp_setup_system - a script for performing "invasive" setup of
     the hardware.  This is *highly optional* for regular users, as it
     will make modifications to the system such as installation of
     system libraries, reconfigure disks for I/O tests, etc.  If the
     user has already set up the system as they wish for the test, this
     script can be skipped.
   * stp_create_test_request - a script for generating a
     request-details file, that includes things like the LILO option,
     Script Parameters, etc.  This step replaces the need to pull the
     test request from the database.  Running this script is optional as
     long as the user already has a request-details file available.
   * stp_verify_system - a script that checks the state of the machine
     and reports any errors.  Verifies things like disk configuration
     (for disk I/O tests), network configuration (for network tests),
     etc. etc.  This script does not cause any changes to the system and
     so is safe to be run on any system.  This allows a user to validate
     his system and perform updates to it manually until it passes.
   * stp_include.sh - a script for setting up some of the initial
     environment variables and for gathering results and logs.  This can
     be called from the test package's wrap.sh script to establish its
     environment.
   * stp_postprocess - a script for postprocessing results and logs,
     generating graphs, etc.
   * stp_upload_results - a script that allows uploading results to a
     repository.  This allows users to package and deliver their results
     back to OSDL, and/or to their own user-defined central repository.
     This is optional and does not need to be used for ordinary
     "one-off" type test runs.

Using the STP-Lite library is straightforward, as essentially it just
adds some environment variables that are passed to the test application
proper.  This provides a language-independent interface for passing data
from the framework down.

When executed, the test will generate a test run ID number for the test
and create a corresponding directory for storing results.


Installing and Running STP-Lite Tests
=====================================
0.  Set up the system, including workloads, applications,
    tools, instrumentation, etc.
    * stp_setup_system may be employed for this, if desired
    * Otherwise, user can set things up manually, and use
      `stp_verify_system` to check their work
1.  Install the STP-Lite package (see its INSTALL file)
2.  Install the test package (see its INSTALL file)
    * Generally, the test package can be run from its build directory
3.  Reboot system with the desired kernel and environment (optional)
4.  Customize the test run
    * Set environment variables (optional)
    * Identify commandline options to pass to `wrap.sh` (optional)
    * Modify configuration files (optional)
    * Create a request-details file.  `stp_create_test_request` can be
      used to generate this, or user can generate their own
5.  Call `wrap.sh` with any commandline options
    5.1  `stp_include.sh` is used to establish the environment settings
    5.2  `stp_verify_system` is called to ensure system is suitable for
         the test being run.
    5.3  The test is installed, set up, and executed as appropriate
    5.4  `stp_postprocess` is called to process results, generate
         graphs, compress large logs, etc.
    5.5  `stp_upload_results` is called to (optionally) deliver the
         results back to OSDL's results repository or to a user-defined
         location.
    5.6  `stp_cleanup` is called to verify the system is prepared for
         the next run.  This is technically not required
6.  Results will be placed in the indicated results directory
7.  Modify system (such as changing kernel) and repeat from step 3 or 4.

Creating STP-Lite Tests
=======================
Wrappering test code for use in an STP-Lite fashion requires several
steps to make it work correctly and to package it in a conformant
manner:

TBD - adapt from existing documentation on creating tests for STP