Test Runner & Logger Framework

A lightweight, deterministic Bash test execution and reporting framework for executing automated regression tests and generating JUnit-compatible XML reports.

Overview

This framework provides a minimal and deterministic infrastructure for running shell-based test cases and producing machine-readable test reports.

It is intentionally explicit:

• no hidden orchestration
• no implicit state mutations
• reproducible execution traces
• deterministic report generation

Design Goals

The framework is designed for environments where:

• tests are plain executable scripts
• no external test harness is desired
• CI integration is required
• execution traces must be reproducible
• debugging must remain transparent

Compatible with:

• Jenkins
• GitLab CI
• Azure DevOps
• any JUnit XML consumer

Architecture Model

The system consists of three strictly isolated roles.

Testcases — Event Producers

Testcases describe behaviour and emit structured events.

They:

• prepare scenarios
• observe behaviour
• emit events
• return execution status

They never:

• write XML
• control persistence
• determine final suite results

Runner — Execution Authority

The runner:

• discovers testcases
• executes them deterministically
• evaluates lifecycle outcomes
• assigns final verdicts
• supervises framework integrity

The runner owns execution semantics.

Logger — State Authority

The logger:

• consumes an ordered FIFO event stream
• maintains the execution state model
• is the single writer of the XML report
• guarantees structural consistency

The logger owns report persistence.

Execution Model

Execution is strictly linear:

testcase → FIFO events → logger → XML state

Properties:

• deterministic ordering
• append-only event stream
• immutable execution trace
• replayable behaviour reconstruction

Event Protocol

The Logger consumes a strict FIFO-based event stream.

Each event is a single line:

EVENT | PARAMETERS...

The protocol is:

• append-only
• deterministic
• transport-only

The protocol defines no semantics.

Semantics are implemented exclusively by the Logger.

Execution Context

Testcase execution is controlled via the init event.

init | <testcase-name> [ | <testsuite-name> [ | <noOfCases> ] ]

Effects:

• creates <testcase> node
• opens a testcase execution context
• routes all subsequent events to this testcase

The logger always maintains exactly one active testcase context.

Orphan Testcase Handling

If an init event is received while another testcase context is active:

• the previous testcase is finalized with status inconclusive
• a suite-level diagnotic warning is emitted
• a new testcase context is opened

Finalization is performed by the Logger as part of context transition handling. This prevents protocol corruption and guarantees continued logging.

Features

Deterministic Execution

• strict testcase ordering
• no parallelism
• no race conditions
• FIFO event processing

JUnit XML Output

• fully valid JUnit XML
• CI compatible
• incremental XML construction
• consistent failure reporting

Blob Support

• arbitrary multiline data
• binary payload support
• stored in CDATA sections

Clear Responsibility Separation

Component	Responsibility
Testcase	behaviour & observation
Runner	lifecycle & verdict
Logger	state & persistence

Minimal Dependencies

• Bash
• xmlstarlet (reference logger)
• sed

Directory Structure

project/
tests/
bin/
  runner
  logger
lib
  bash-test-framework
    api.sh
    assert.sh
share
  bash-test-framework
    example
      testcase-blackbox.tcase
      testcase-whitebox.tcase
    tests/
      runAllLogger.sh  # execute all logger/*.sh scripts
      logger
        testscripts.sh
      example        # mini testsuite example
        runSuite.sh  # run all testcase scripts in suite testSuite
        testee
          urcMath.sh # whitebox testee with Unique Return Codes
          Math.sh    # blackbox testee
        testSuite
          01-div/*.tcase    # whitebox tests of div() function
          02-main/*.tcase   # whitebox tests of main() fucntion -> CLI
          03-integration/*.tcase   # blackbox tests of productive tool
          runEach.sh        # execute all ??-* subnodes in a row

Testcases matching:

NN*-name.tcase

are executed in sorted order.

Writing Testcases

A testcase is a normal executable shell script.

Requirements:

• no invocation parameters
• deterministic behaviour
• must not corrupt workspace state

Testcase Public API

The public API guarantees:

• deterministic execution semantics
• replayable event traces
• stable result model
• strict isolation from persistence

Testcases must not access:

• XML files
• logger internals
• framework state objects

Framework Home Resolution

The framework root directory is exposed via:

__tc_framework_home

Resolution order:

(1). Environment preset

export __tc_framework_home="${BASH_TEST_FRAMEWORK_HOME:-}"

(2). Automatic discovery heuristic

Standard installation locations are searched.

(3). Failure

Execution aborts if no installation can be resolved.

The resolved value remains immutable for the runner lifetime.

Execution Result Model

Testcases return symbolic RC values:

RC	Meaning
RC_PASSED	behaviour correct
RC_FAILED	functional violation
RC_ERROR	infrastructure problem
RC_SKIPPED	testcase intentionally skipped

The runner assigns the final verdict.

Logger Write Modes

Write modes control persistence frequency only.

full

XML written after every event.

verdict

XML written after testcase completion.

final

XML written once after suite completion.

Running a TestSuite

A testsuite is simply a directory structure containing .tcase files.

Example:

runner --onError=continue regression

Options:

--suiteName={name to be used in log file} : If omitted, the suiteName is derived as follows:

• If exactly one input argument is provided and it is a directory: the suiteName is the directory name.

• If multiple input arguments are provided: the suiteName is "selection-" followed by a deterministic hash computed from the input arguments in the exact order given.

The hash uniquely represents the invocation parameters and ensures stable suite identification across identical runs.

--onError=continue : Continue executing testcases even if errors or failures occur. (Default)

--onError=halt : Stop after the first test case reporting ERROR or FAILED

--writeMode={full|verdict|final} : logger persistence mode

Runner Behaviour

The runner automatically:

• discovers testcases
• starts logger
• executes testcases sequentially
• assigns verdicts
• protects logger integrity
• waits for final XML completion

Logger Output

The following representation describes the logical XML tree structure.

It is not a literal XML serialization but a simplified hierarchical view comparable to XPath navigation.

Simplified structure:

testsuites/
  testsuite/
    system-out/
    system-err/
    testcase/
      system-out/
        objective
        description
        references/
          reference
        preconditions/
          precondition
        postconditions/
          postcondition
        steps
        assets
        blobs
      system-err/
    testcase/
    ...

Logger Implementations

Bash Logger — Reference Implementation

• transparent
• event-by-event XML updates
• ideal for debugging

Not optimized for performance.

Python Logger — Production Implementation

• in-memory state model
• efficient XML generation
• optimized for large regression suites

Recommended for CI usage.

---

Design Principles

• explicit communication
• append-only execution model
• deterministic behaviour
• single writer persistence
• failure isolation
• long-term maintainability

The framework intentionally avoids abstraction layers that obscure execution behaviour.