TestReportDescription

bstanly

Sample Test Report Generator

  Author: Barry Stanly Copyright 2015 by Barry Stanly June 25, 2017

C O N T E N T S

1 Introduction
1.1 Organization

2 File Formats
2.1 ProjectStructure
2.2 ProjectDescription
2.3 ProjectRequirementsMap
2.4 ProjectTestResults

3 Formal Test Report
3.1 Conditional File Inclusion
3.2 Controlling Font Size
3.3 Using Landscape Mode
3.4 Combining With Other Material

1 Introduction

In reading this document, Familiarity with the STF Syntax Summary is assumed. (Download document PDF)

This document describes the Sample Test Report Generator (testreport.py) included as an example of how to use the Python interface (stflib) to the Simple Text Formatter (STF) automated document generation system. Although it is an example, it is intended to be useful and may be customized as needed.

It is designed to produce a quick look test report. The quick look report shows all available information in a concise, but unsophisticated format. However the tables generated by the Test Report Generator (TRG) are also available to be included in more formal test reports. Accordingly a sample formal test report template designed to use the tables produced by the TRG is also included. The formal test report is suitable for customization so that a formal test report can be automatically generated along with the quick look test report.

Because the quick look report produces the necessary tables for the formal report, the quick look report will be described first.

Permission is granted to copy, distribute and/or modify this document under the terms of the
GNU Free Documentation License, Version 1.3 or any later version published by the Free
Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.
A copy of the license is available at: https://www.gnu.org/copyleft/fdl.html.
For more information, see the STF Syntax Summary


1.1 Organization

The test report script, hereinafter referred to as TestReport, operates off of a set of files. Each file name starts with the project name, hereinafter referred to as Project. The second part of the file name is fixed. The files and their usage are as follows:

  1. ProjectStructure.txt -- this is the control file. It defines the output directory and format.

  2. ProjectDescription.txt -- this is a description of all the test levels. Typically this includes, subsystem, suite, and test. The designations are purely arbitrary and are defined in ProjectStructure.txt. At least one test level is required.

  3. ProjectRequirementsMap.txt -- this is optional. If included it specifies what requirements are tested by each test. Each test may test more than one requirement and each requirement may be tested by multiple tests.

  4. ProjectTestResults.txt -- this contains the test results.All files may specify include files. When an include file reference is encountered, processing continues with the include file until it is exhausted, then processing continues from the next line in the parent file (the file that held the include file reference.) Include files are specified as:
    ->path starting in column 1.
    Include files may be nested.


2 File Formats

General:

  1. A pound ("#") sign in column 1 indicates a comment. Comments are ignored.

  2. Lines less than three characters long are ignored.

  3. Lines starting with "->" are interpreted as an include file reference.

  4. A line ending in a backslash ('\') is automatically concatenated with the following line, making in effect one long line.

  5. A double colon is used to separate test levels. For example, assume three test levels: Subsystem, Suite, Test. Further assume a subsystem name of "Satellite", a test suite name of "Broadwing", and a test name of "Signal". Then the full test name would be: "Satellite::Broadwing::Signal"

  6. Table column formatting conventions are borrowed from LaTeX(1). These are discussed in the STF Syntax Summary. The column formatting specification is referred to as the cspec.

  7. \t is used to represent a <TAB> character (Hex 09).The individual file formats are shown in the following paragraphs.


2.1 ProjectStructure

A sample structure file is as follows:

# Sample Test Structure
Project=SkyWhen
Structure=Subsystem,Suite~~~~,Test~
Description=Division of SkySky\t\tfrom a Good Company\t\tSample Test Report
PrintResults=True
ReportDir=./report
TestReport=True
Requirements=True
Xref=True
PIECharts=True
Format=Description:|l|l|l|p{3in}|

These fields are interpreted as follows:

  1. # Sample Test Structure -- A comment, which is ignored.

  2. Project=SkyWhen -- This sets the project name to "SkyWhen".

  3. Structure=Subsystem,Suite~~~~,Test~ -- This defines the test levels. There must be at least one test level. There is no limit (other than the project manager's sanity) to the number of test levels. A more complicated case would be "System,Suite,Test,Case", this defines four test levels. The least complicated case would be "System", this defines one test level. Tildes ("~") are used to expand the implied tabular column length. The column sizes may be specified by the length of the test level names in the Structure directive; or specified in a column specification, or cspec, in a Format directive. If no cspec is given, defaults based on implied column length in the Structure are used.

  4. Description=Division of SkySky\t\tfrom a Good Company\t\tSample Test Report -- This sets the document title of the quick look test report. Tabs ("\t") indicate blank lines in the title page.

  5. PrintResults=True -- This turns on the diagnostic printout in the document log. False turns it off.

  6. ReportDir=./report -- This defines the output directory to hold the test report files. The default is the current directory.

  7. TestReport=True -- This specifies that a test report is to be produced. False is used when only diagnostic output is desired.

  8. Requirements=True -- This specifies that a requirements map is included. The test report will then include a table giving the requirements test status.

  9. PIECharts=True -- This specifies that Gnuplot(2) pie chart script files are to be generated for the requirements status and the test status. The Gnuplot requirements pie chart script file is named PieChartRequirements.gp; the test status pie chart script file is named PieChartTestSummary.gp. Invoking Gnuplot on these files causes PieChartRequirements.jpg and PieChartTestSummary.jpg to be produced.

  10. Xref=True -- If Requirements are being processed, then a cross reference between tests and requirements will be produced showing which tests, test what requirements; and which requirements are covered by what tests.

  11. Format=Description:|l|l|l|p{3in}| -- This is a typical cspec. The format of the 'Format' directive is:
    Format='Table-Name':cspec
    This particular cspec applies to the Description table and specifies four columns. The first three columns are left justified and adjust to the field size without wrapping, the fourth column automatically wraps to three inches. The table names are as follows:

   a. Description -- Specifies column width for the description table. There is one column for each test level plus an additional column for the description. The sample had three test levels, so there would be 3 + 1 or four columns in the example.

   b. Requirements -- If Requirements is specified, then the requirements test status will be generated. The default cspec is "|l|c|c|c|", specifying an initial column for the requirement name and an additional three columns for "Passed", "Failed", and "Not Tested".

   c. RTXREF -- This specifies the cspec for the "Requirements To Test" cross reference. The default cspec is "|p{1.5in}|p{5in}|".

   d. TRXREF -- This specifies the cspec for the "Test To Requirements" cross reference. The default cspec is "|p{5in}|p{1.5in}|".

   e. "Test-Level-Name" -- Each test level represents a table in a test report. The first test level needs three columns; the second test level needs four columns and so on up until the last test level. The last test level needs a column for each level plus one to hold the results. So, to use the example, there is a table for subsystem test status consisting of three columns; for the test suite level the table consists of four columns; and for the last level, the test level, the table needs four columns.


2.2 ProjectDescription

A sample Project Description file is as follows:

# SkyWhen test description
# Project=SkyWhen
# Structure=Subsystem,Suite,Test

Part-A::This is a good subsystem part

Part-A::Test Suite 1::This is test suite 1, subsystem A
Part-A::Test Suite 1::Test 1::This tests something - A
...
Part-A::Test Suite 1::Test 6::and still more - A

Part-A::Test Suite 2::This is test suite 2, subsystem A
...

Part-B::This is an even even better subsystem part

Part-B::Test Suite 1::This is test suite 1, subsystem B
...

The file is interpreted as follows:
Each test level is listed, including the tests along with their descriptions. When the test results are interpreted, each test result is matched against a corresponding test description in this file. That way any tests that are omitted from the test result file will show up as "Not Tested".

Also the test description table will list every subsystem, suite, and test along with the corresponding description.

Tests that show up in the test results file that are not in the description will be included in the test results. However tests that should be included in the test results file that are not in the results file nor in the Project Description file will, obviously, not be shown.


2.3 ProjectRequirementsMap

A sample Project Requirements Map file is as follows:

# Sample Requirements Map
# Project=SkyWhen
Part-A::Test Suite 1::Test 1::ReqId01, ReqId02
Part-A::Test Suite 1::Test 2::ReqId03, ReqId04, ReqId02
Part-A::Test Suite 1::Test 3::ReqId05, ReqId06
Part-A::Test Suite 1::Test 4::ReqId07, ReqId08, ReqId01
Part-A::Test Suite 1::Test 5::ReqId09, ReqId10
Part-A::Test Suite 1::Test 6::ReqId11, ReqId12
Part-A::Test Suite 2::Test 1::ReqId03, ReqId04, ReqId01, ReqId02
Part-A::Test Suite 2::Test 2::ReqId15, ReqId16

The format is to list the full test name followed by a comma separated list of requirements that the test, tests.


2.4 ProjectTestResults

This file contains test results. Sometimes it is advantageous to make this file a container file with include file pointers to the real test results data. That way it is easy to accommodate new test results. In case of the same test status being specified more than once, the last value read takes precedence. A sample file is shown below:

# Sample Test Results
# Project=SkyWhen
# Structure=Subsystem,Suite,Test
Subsystem=Part-A
Suite=Test Suite 1
Test=Test 1::Passed
Test=Test 2::Passed
Test=Test 3::Passed
Test=Test 4::Passed
Test=Test 5::Failed
Test=Test 6::Failed
...
Full=Part-B::Test Suite 2::Test 6::Passed

It may be seen that there are two formats that may be used to specify test results: A test level based format where each level is shown on its own line; and a full format which shows the full test name followed by the test result.


3 Formal Test Report

The quick look report uses a description string to identify the report. This is suitable for informal documentation. A formal test report uses a cover page format. Cover pages appear before the table of contents, usually are unnumbered, and specify various forms of identification as required by the company or customer receiving the report. The example uses a simple cover page as an example. Fix it up to correspond to your organization's standards.


3.1 Conditional File Inclusion

The Formal Test Report (FTR) is setup to include some files conditionally. For example the requirements test status and requirements cross reference may be omitted. If so, it would be an error to try and print them. The way conditional inclusion works is a label variable is defined as follows:

.Activate %Exist(Suite:SkyWhenSuitetests.sti)
This defines label "Suite" if file SkyWhenSuitetests.sti exists. The label is then used to activate a conditional section in the document as follows:
.Ifc Suite
The test suites are defined in the master test plan, see reference 27...
.Include SkyWhenSuiteTests.sti
.endc


3.2 Controlling Font Size

It may be advantageous to reduce the font size for a table so it will look better in the formal test report. This usually entails two actions:

  1. Define the cspec for the table so that the shrunken table will span the full width of the paper.

  2. Shrink the font in the formal test report so that more characters fit on each line.One way to shrink the font is to use the \small{} macro, defined in the STF standard macro library (StfMacros.stf). This appears as follows:

    \small{
    .Include SkyWhenSuiteTests.sti
    }

    This shrinks the font somewhat so that more information will fit on a line.


3.3 Using Landscape Mode

Another way to manage wide tables is to switch to landscape mode for the table(s). It is also necessary to set the cspec to stretch the table to fit the wider page. This looks like the following:

\landscape{begin}
.Include SkyWhenSuiteTests.sti
...
\landscape{end}

This will cause the page(s) for the above material (tables plus other verbiage) to be shown in landscape mode.


3.4 Combining With Other Material

There are three standard ways to combine an STF document with other material:

  1. Change the cover page(s) to mark the output as an attachment and stick the automatically generated report at the end of another document.

  2. Incorporate additional material in STF format and include it into the formal test report as separate sections. Typical examples are: the conclusion section, test synopsis, other required boilerplate, etc. The STF processor, STFXlate, will merge all its inputs into a TeX file, which can be processed by pdflatex to create a PDF format report.

  3. Choose to output to RTF format (see the STF Syntax Summary for how to do this) and manually merge the information into a MS Word format document. Sometimes this is a useful approach when conforming to company requirements.

F O O T N O T E S



(1) For example, see http://www.cs.cornell.edu/info/misc/latex-tutorial/introduction.html

(2) Gnuplot may be downloaded from http://www.gnuplot.info/download.html.

Related

Wiki: DocumationMagic