Operation Mix is the name given to a set of operations that will be run as part of a test. A mix consists of one/more operations which may be comprised of the standard built-in operations or user defined custom operations.
The loading of mix files is actually designed to be entirely configurable, currently two mix formats are supported:
The Classic format is the text based format supported by the old 1.x API which only supports fixed query operations. This format is a simple line based text format whose files have a .txt extension. Each line is an absolute/relative path to a file containing a query to be run e.g.
queries/1.rq queries/2.rq
The 2.x format was introduced in the 2.0.0 release and is a line based TSV format whose files have a .tsv extension. It supports loading any of the built-in operations and any user defined custom operations. Each line in the file consists of several tab separated arguments where each line describes the information necessary to load one specific operation e.g.
query queries/1.rq Query 1 (ASK) update updates/1.ru DELETE rdf:type Triples sleep 10 10 Second Sleep
In the above example we create a mix consisting of several different operation types, different operations will require different arguments and these can be examined by using the operations command in the CLI or by calling the getArguments() method of the appropriate OperationLoader implementation.
Each line of the mix file has an operation name in the first field followed by arguments for configuring that operation in subsequent fields. Some arguments may be optional and may be omitted in which case default values for those arguments are used.
You can add support for a custom mix file format by implementing your own OperationMixLoader class and registering it with the OperationMixLoaderRegistry. Each loader is associated with a particular file extension so you should ideally choose something that does not clash with the built-in mix file formats though you may wish to override the default loading logic for those formats.
The following section gives a brief guide to some of the most commonly used operations, you should use the operations command to view the CLI built-in help for the full range of operations.
Where configuration examples are given they are in the 2.x TSV format.
There are several different variety of query operations supported all of which make a SPARQL query with some small variations in how the query is made.
The simplest operation is the query operation (FixedQueryOperation in the API), this operation runs the same SPARQL query every time it is run and is primarily used when you want to gauge performance for a specific query.
A slightly more powerful operation is the param-query operation (ParameterizedQueryOperation in the API), this operation runs a SPARQL query where each run of the query substitutes a set of parameters into the query. This allows you to gauge performance of a query template so to speak and may help partially defeat caching that the query engine may be doing to improve performance. This operation is configured like so:
param-query queries/1.rq params/1.tsv Parameterized Query 1
Here we specify both a file for the query and a file for the parameters. The parameters file is expected to be a SPARQL results file in the SPARQL Results TSV serialisation where each line in the file represents one possible set of parameters. Each time the operation is run then a set of parameters is selected from those available at random and substituted into the query before it is sent to the SPARQL system for execution.
Both of this operations have a variant which allows you to add custom name value pairs to the requests in order to test custom/extended behaviours of the system being tested. These are the nvp-query and param-nvp-query operations respectively, they are configured like so:
nvp-query queries/1.rq params/nvps.tsv NVP Query 1 param-nvp-query queries/1.rq params/1.tsv params/nvps.tsv Parameterized NVP Query 1
These operations take an additional argument compared to their non-nvp versions which is a file containing the name value pairs. This file may either be a text based Java properties file in which case it must have a .properties extension or be a TSV file with a .tsv extension. If the Java properties format is used then each custom parameter can only be given a single value. However the TSV format takes the first field of each line to be the custom parameter name and then all subsequent fields on that line to be values for the parameter so you can have multiple values for a parameter if that is desirable.
These operations operate on remote SPARQL services, there are also a mem-query and mem-param-query variants which instead operate over in-memory datasets instead. See In-Memory Testing for more information about how in-memory testing works.
Similar to query there are several different varieties of update operations supported all of which make a SPARQL update with some small variations in how the update is made.
The simplest operation is the update operation (FixedUpdateOperation in the API), this operation runs the same SPARQL Update every time it is run and is primarily used when you want to gauge performance for a specific update.
A slightly more powerful operation is the param-update operation (ParameterizedUpdateOperation in the API), like the query equivalent this runs a SPARQL update where each run of the update substitutes a set of parameters into the update. This allows you to gauge performance of an update template so to speak, the operation is configured like so:
param-update updates/1.ru params/1.tsv Parameterized Update 1
Like the equivalent query operation the parameters file is expected to be in SPARQL Results TSV format with each result representing one possible set of parameters.
As with the query operations there are nvp variants of the update operations as well - nvp-update and param-nvp-update. This are configured just like their NVP query equivalents and use the same NVP formats as those operations support.
These operations operate on remote SPARQL services, there are also a men-update and mem-param-update variants which instead operate over in-memory datasets instead. See In-Memory Testing for more information about how in-memory testing works.
The sleep operation (SleepOperation in the API) does exactly what the name suggests, it sleeps for some period of time. This is useful since it allows you to simulate periods of inactivity which may be particularly relevant for soak testing though less so for benchmarking.
The operation is configured like so:
sleep 30 Sleep for 30 seconds
The configuration consists simply of the time you wish to sleep in seconds and optionally a friendly name for the operation. Note that when operation timeouts are used the sleep time may not exceed the configured operation timeout.
There are other operations besides those detailed here, you can use the CLI operations command to view help for the available operations including detailed help on configuring each type of operation in your mix file.
The API is designed to allow for injecting custom operations into it as desired, implementing a custom operation typically requires implementing 2-3 classes depending on what your operation will do.
Firstly you need an implementation of the Operation interface, in most cases you will probably want to extend AbstractOperation since that will give you some parts of the implementation for free. The first things you want to implement and the descriptive methods getType() and getContentString() which provide information about the operation that will be displayed to users when running the operation. Then you will want to implement the canRun() method, this method returns true or false depending on whether the operation can run with the given Options. For example if you were writing a query operation then you would likely want to enforce that the query endpoint has been set.
The real key method of this interface is the createCallable() method, this creates a OperationCallable which encapsulates the work of actually performing the operation and returning results for a single run of it. Depending on what your operation does there may already be a pre-built callable e.g. QueryCallable runs a query or NvpQueryCallable runs a query with some NVPs added to the request. However if you need to implement your own again there are various abstract classes like AbstractOperationCallable that will give you parts of the implementation.
Once you have your Operation and OperationCallable implementations ready you then need to create an OperationLoader that is responsible for actually loading instances of your operation from an operation mix file. In most cases you'll probably want to start by extending the AbstractOperationLoader and you can then start implementing methods. There are several descriptive methods - getPreferredName(), getDescription() and getArguments() that you will likely want to implement first. These provide information about how an operation is configured, getPreferredName() returns the operation name that is used to refer to this operation in mix files while getDescription() returns a brief description of what the operation does. The getArguments() method returns an array of OperationLoaderArgument instances which describe the configuration arguments your loader expects to receive.
The method that does the actual work is the load() method, this takes in the base directory (for resolving relative paths) and the array of arguments for the operation (which excludes the operation name). Here you will implement whatever logic is necessary to create an instance of your Operation implementation from the given arguments, if there are incorrect/insufficient arguments you should throw an IOException.
Finally you can now register your custom operation loaders with the API, to do this simply call the addLoader() method like so:
OperationLoaderRegistry.addLoader(new MyCustomOperationLoader());
This will register your operation using its getPreferredName() method to choose the operation name used to refer to the operation. In some cases it may be desirable to register your operation until multiple names in which case you can do the following:
OperationLoader loader = new MyCustomOperationLoader();
OperationLoaderRegistry.addLoader("a", loader);
OperationLoaderRegistry.addLoader("b", loader);
And if you ever want to remove a loader you can do so with the removeLoader() method which takes the name of the operation for which you want to remove the loader. You can also go back to only the built-in implementations by calling the resetLoaders() method.
