MatlabbatchDevelopmentBatchUI

Non-interactive setup of jobs

The matlabbatch system originated from SPM5 spm_jobman.m. Although it has been generalised and some more command line functionality has been added to its current incarnation, non-interactive manipulations of batch jobs are only supported to a limited degree.

Using job templates and BasicIO

The recommended usage scenario is to create study-specific processing pipelines using the GUI. These pipelines can be templates with some settings left empty. Obviously, this does only work for cfg_entry, cfg_files or cfg_menu items with a value <UNDEFINED> in the job template. Once the templates are created, there are easy-to-use command line calls to fill the <UNDEFINED> values for this kind of templates.

It does also work to set cfg_repeat or cfg_choice items, but only if they are <UNDEFINED> and require at least one item to be set. However, one can't change predefined values in the resulting job.

One rationale behind this is that the batch system should do all the scheduling, job execution, dependency resolution etc. for the whole processing pipeline, not just for each processing step separately. Another rationale is that the job templates should serve as a kind of documentation what has been done to the data.

The following set of cfg_util callbacks allows to initialise a predefined job, fill it and run it:

cfg_util('initcfg')                           % Initialise batch system
jid = cfg_util('initjob', myjob)              % Initialise job
cfg_util('filljob', jid, input1, input2, ...) % Fill in missing values
cfg_util('run',jid)                           % Run job

Also, BasicIO modules allow a great deal of flexibility. In many cases, one can construct batch templates which only take a single folder as argument and do all file selection stuff etc using regular expressions. Of course, this is limited to a fixed job structure, where no cfg_repeat or cfg_choice items need to adapted to the actual data on disk. If there is a clearly defined file system hierarchy, one could write a batch module that takes the directory of a single dataset as an input and returns all available data as dependencies. But again, the expected outputs have to be selected by the user/batch configurator before the module is run (i.e. without any reference to actually existing data). Missing or non existent data in a real subject can not be detected while the batch is being created.

Programmatic modification of a job

The GUI cfg_ui uses callbacks of cfg_util to modify jobs. If a collection of job templates is not flexible enough to deal with the data, the solution would be to write a program that replaces cfg_ui. While cfg_ui is usable for all batch-enabled applications, it is most likely, that such a replacement program has to be written specifically for each module, because it needs to hard-code details about the job structure of this module. This actually defeats a key purpose of the matlabbatch system - to hide away the complexity of configuration details from users and application programmers. In the batch system, the only place where configuration details should go is into the configuration files.

However, if such a tool is really necessary, the following code and comments will give a short explanation how to program it. A good starting point is to analyse a saved job. The field names of the struct/cell job datastructure are needed to address executable modules (cfg_exbranch) and required input fields. The job management in cfg_util always uses some kind of id data to reference jobs, modules, input items. One should never try to modify these id variables outside cfg_util.

The following example requires SPM8 to run. Similar code can be used for BasicIO or other applications. The only modification would be the use of valid tag strings and the supply of valid inputs (this is where information from the application config file needs to be hardcoded in the batch setup utility).

First, we initialise an empty job:

jid = cfg_util('initjob');

Then, we ask for the ids of the quality input in the SPM realign.estimate default configuration:

[mod_cfg_id, item_mod_id] = cfg_util('tag2cfg_id', 'spm.spatial.realign.estimate.eoptions.quality');

This returns an id for the spm.spatial.realign.estimate module and the id for the eoptions.quality input within that module. One can then use these ids to add the estimate module to the job, and to set the eoptions.quality field:

mjid = cfg_util('addtojob',jid,mod_cfg_id);
cfg_util('setval',jid,mjid,item_mod_id,.1);

This would set the registration quality to .1. In a similar way, one can set file or menu inputs. For cfg_repeat/cfg_choice items, this becomes more cumbersome. To add e.g. a session of data to the above job, one has to run:

[mod_cfg_id, item_mod_id] = cfg_util('tag2cfg_id','spm.spatial.realign.estimate.data');
cfg_util('setval',jid,mjid,item_mod_id,[1 1]);

This would return the id for the data input field. Then, the first (and in this case only available) option for the repeated data would be set to be the first item in the job. While cfg_ui retrieves the necessary information from cfg_util and leaves the decision to the user, a more specialised tool would probably require to read the corresponding config file and hardcode this. To add another session, one would do

cfg_util('setval',jid,mjid,item_mod_id,[1 2]);

Then, you will have to get the item_mod_ids for the newly added sessions.

[item_mod_idlist, stop] = cfg_util('listmod', jid, mjid, item_mod_id, cfg_findspec);

You can be more specific and specify a cfg_findspec to look only for cfg_files, a specific tag or GUI label etc. or even specify configuration traversal options to only return immediate children (see cfg_tropts). Once you've got the item_mod_idlist, you can use it to assign data to the sessions. The item_mod_idlist entries will always be relative to item_mod_id, not to the module. To set both data items, one would do:

cfg_util('setval',id,mjid,[item_mod_id item_mod_idlist{2}],{'/tmp/test1.nii','/tmp/test2.nii'})
cfg_util('setval',id,mjid,[item_mod_id item_mod_idlist{3}],{'/tmp/test4.nii','/tmp/test3.nii'})

If for some reason cfg_util('setval',...) leaves the job unchanged, it means that you tried to assign something wrong...

After you've set up the input structure of a module (remember: you don't need to fill in file names/variable values yet), you might want to harvest it:

cfg_util('harvest',jid,mjid);

If the module input structure is complete, this will set up virtual outputs for this module. You can get them with

voutputs = cfg_util('getallvoutputs',jid,mjid);

This will be a cell with one element - an array of cfg_dep dependency objects, describing the expected output structure of this module. When another module is added to the job, one can use cfg_util('setval',...) to place any of these dependencies into the inputs of matching items. The order and contents of these cfg_dep arrays is defined by the .vout function of each cfg_exbranch. The order and meaning of the virtual outputs is again described in the application configuration files.