API

Introduction

The framework is designed such that the CLI is merely a shim over the underlying API, this means that you can easily write your own custom CLIs or run testing purely through programmatic calls to the API.

The easiest way to integrate the API into your code is by adding a dependency on our Maven Artifacts, to use the API then you will need to add a dependency on the core artefact.

API Overview

The API is designed to separate the different concerns as much as possible to make it easy to extend one part of the system without needing to touch others. The major concepts of the API are as follows:

• Options - The Options interface and its implementations provide the configuration information for testing.
• Operations - The Operation interface and its related interfaces and implementations encapsulates the operations to be tested
• Runners - The various runner interfaces encapsulate the logic of actually running tests
• Progress Listeners - The ProgressListener interface provides a mechanism to monitor the ongoing progress of testing

You can also browse the javadocs for the API.

Options

Options is a key interface in the API, it provides all the common configuration options that you might want to provide to an operation or to a runner. OptionsImpl provides a concrete implementation of the common options and there are further sub-classes which provide runner specific options.

To run any kind of test you will need to create an Options instance and populate it appropriately e.g.

BenchmarkOptions ops = new BenchmarkOptions();
ops.setRuns(250);
ops.setWarmups(10);
ops.setQueryEndpoint("http://example.org/query");

What options you must set will depend on the runner you wish to use, runners provide a canRun() method which will return true if all required options are set and false otherwise.

Operation Mixes

The other option you will always need to set as well as your general options is the operation mix via the setOperationMix() method. This takes in an instance of the OperationMix interface which you may create in a couple of ways:

• Loading from a file
• Creating it programmatically

Loading Operation Mixes from files

As discusses on the Operation Mixes page we support file based formats for configuring operation mixes. If you wish to simply load a prepared operation mix file you can do this like so:

// Firstly try and get a mix loader based on the known file extension
OperationMixLoader loader = OperationMixLoaderRegistry.getLoader("tsv");

// Check it for null
if (loader == null) throw new RuntimeException("No mix loader available for files with the given extension");

// Use it to load the mix
OperationMix mix = loader.load(new File("example.tsv"));

// Finally set it on our options
ops.setOperationMix(mix);

Creating Operation Mixes programmatically

Alternatively you can create an OperationMix entirely through the API e.g.

// Create a list and fill it with operations
List<Operation> ops = new ArrayList<Operation>();
ops.add(new FixedQueryOperation("Query 1", "ASK WHERE { ?s a ?type }"));

// Then set it on our options
ops.setOperationMix(new OperationMixImpl(ops));

Setup and Teardown Mixes

In addition to the main operation mix you can also specify separate setup and teardown mixes which run before and after the tests. These are guaranteed to be run exactly in order whereas the main operation mix can be configured to run in random order or have its running otherwise customised.

Operations

The API has a range of different built in operations already provided and is designed to allow you to easily add your own, please see the Operation Mixes documentation for more details on creating custom operations.

To see the available operations take a look at the Operation interface javadoc or use the CLI operations command.

When creating operations programmatically it is usually a simple case of just calling the appropriate constructor as seen in the earlier example.

Runners

Runners are the classes responsible for carrying out actual testing, there are actually three separate runners interfaces each with different responsibilities:

• Runner - This interface is responsible for the high level running of tests and provides a means for operations run within the tests to report their progress
• OperationMixRunner - This interface is responsible for running an operation mix and returning a OperationMixRun which describes information about that run.
• OperationRunner - This interface is responsible for running an individual operation and returning an OperationRun which describes information about that run.

Runner

The API contains an abstract implementation AbstractRunner and two concrete implementations:

• BenchmarkRunner - Runs a benchmark
• SoakRunner - Runs a soak test

The abstract implementation provides the generic parts of the interface implementation e.g. progress reporting as well as useful helper methods for doing things like running setup and teardown mixes.

The heart of the Runner interface is the run() method which takes in a set of Options and runs the actual tests. So to run the benchmark we prepared the options for earlier we'd do the following:

BenchmarkRunner runner = new BenchmarkRunner();
runner.run(ops);

OperationMixRunner

The OperationMixRunner interface is responsible for carrying out runs of an operation mix, typically this means deciding what order to run the operations in and then using the OperationRunner configured on the Options to run the actual operations.

There is an AbstractOperationMixRunner implementation which gives you the majority of the implementation with derived classes just needing to implement a few protected classes to customise the behaviour as desired.

The API contains several pre-built mix runners:

• DefaultOperationMixRunner - Runs each operation in the mix once and only once, operations run either in random order or in the order specified depending on the provided Options
• InOrderOperationMixRunner - Runs each operation in the mix once and only once precisely in the order they are specified in
• SamplingOperationMixRunner - Runs a sample of the operations in the mix where the sample size may be smaller/larger than the operation mix and optionally contain repeats of the operations i.e. an operation could run more than once in a single mix run.

Just like the other runner interfaces the key method is the run() method which takes in the Options and the OperationMix to run and returns the OperationMixRun containing the statistics for the run. Generally speaking you will not call this directly since invoking this will be handled by the Runner implementation you are using.

OperationRunner

The OperationRunner interface is responsible for running a single operation. Currently there is only a single implementation which is the DefaultOperationRunner which simply tries to run the operation and then returns the OperationRun with the results of the operation.

In the future we plan to add more implementations of this that will allow things like running the operation multiple times and picking the best result, retrying failed operations etc.

Progress Listeners

The ProgressListener API allows for you to implement listeners that are able to monitor the progress of testing to either report on it or to gather statistics. The API provides two main categories of implementation of this interface:

• Progress monitors
• Statistic gatherers

Progress listeners are registered by calling addProgressListener() on your Options instance. Typically you will want to register at least one listener as otherwise you will not be aware of the progress of your tests.

Progress Monitors

The progress monitor implementations are concerned only with reporting the progress of tests, in this way they act very much like loggers. There is an abstract StreamProgressListener which provides the basic implementation for this and then several concrete implementations:

• ConsoleProgressListener - Reports progress messages to standard out
• ConsoleErrProgressListener - Reports progress messages to standard error
• FileProgressListener - Reports progress messages to a file

Statistics Gatherers

The statistic gatherer implementations are concerned only with amalgamating the statistics and outputting them in formats suitable for further analysis with other tools. There are two implementations of this currently available:

• CsvProgressListener - Creates a CSV file containing detailed statistics and run option information
• XmlProgressListener - Creates an XML file containing detailed statistics and run option information

Creating custom progress listeners

Creating custom implementations is relatively straightforward and simply requires you to implement the ProgressListener interface.

Custom implementations can be used to do all sorts of things, internally at Cray we use them to monitor resource usage on the systems being tested and to add additional progress information to make it easier to match test runs up with system logs.