Your First Plug-in Part 2: Using the Job Profile Editor to create Job Launchers

WARNING! - There are several outstanding bugs that may prevent the instructions on this page from resulting in a correctly functioning JobLauncher. We are aware of the problems. The bugs are in our bug tracking system and being squashed!

This tutorial series is for anyone who needs to extend NiCE, for whatever reason, to perform custom tasks such as launching new types of jobs or creating new types of input files. In the first part, Your First Plug-in Part 1: Extending NiCE with custom Items, you learned about the architecture of NiCE and how it performs tasks. You also learned the steps necessary to add a new plug-in to NiCE. This part continues the tutorial with a discussion of the Job Profile Editor that can be used to automate the plug-in creation process for launching jobs.

Table of Contents

• Launching a Job with NiCE
• The Job Profile Editor
• Launching your new Job Launcher
• Next Steps

Launching a Job with NiCE

NiCE is pretty good at launching jobs (also "executing codes," "running jobs," etc.) on local and remote machines. It can even be set up to launch jobs locally or remote in sequential, "sequential and chained" and parallel (launching all the jobs at once) modes via the Multilauncher.

It is very simple to launch a job with NiCE and there are normally only a couple of decisions required: where the job should be launched and its input file. Many codes also support a variable number of MPI processes or threads for parallelism. An example of a Job Launch plug-in for NiCE is shown below for the Ultimate Simulator of Ultimate Destiny! (USUD) test code. This launcher was actually created using the process that is described below, so keep it in mind.

The Job Profile Editor

Launching jobs is so common and simple that many users eventually want to add jobs of their own or customize the ones that are already available in NiCE. As you may have noticed in the last part of this tutorial, it is not a terribly difficult task to add a new plug-in to NiCE directly with Java code.

The Job Launchers in NiCE (at least those developed by the NiCE team!) do not directly inherit from the Item class as described in the last tutorial. There is a subclass of Item called JobLauncher that can launch jobs on local or remote systems and provides many convenience operations and security checks to boot. Developing specialized subclasses of JobLauncher for your first plug-in will be covered in Your First Plug-in Part 3: Creating a Java-based Job Launcher with an OSGi NiCE plug-in. However, since there are so many codes to be launched and they all do it in pretty much the same way, the NiCE team developed a special Item in NiCE that can generate Job Launch plug-ins automatically, the Job Profile Editor. The Job Profile Editor does not write Java code to add a new launcher to NiCE. Instead, it instantiates a JobLauncher, uses the convenience operations to load up the class with all of the details about the job (name, executable, parameters, etc.) and serializes that launcher to disk in a special file that NiCE can read and bootstrap back into a JobLauncher after restarting it. This is all done with NiCE's Serialized Item System, (but that does not matter much for the rest of the discussion). Using the Job Profile Editor is very straight forward. Start NiCE and Create an Item,

and choose Job Profile from the pop-up list,

A new Job Profile will be created and displayed. Do not worry about the number 28 or others in the pictures. Those are only there because that is the 28th Item in my personal database. Yours may be 1, 2 or something else entirely.

The Job Profile is the aforementioned special file that NiCE writes to disk. The editor shows all of the different options that can be used with the launch. The options are not discussed in detail here because they are self-describing: just hover your mouse over one and a text box will appear describing what that option does. If you are feeling brave, you should try to follow this guide and use one of your own test codes. We will create a Job Launcher using the ls shell command (Linux & Unix) as a test code to list the contents of the working directory. If you are on Windows, we recommend using the dir command to accomplish the same task. This is not the most rigorous way to demonstrate the capability to create a Job Launcher, but it will give you the basic feel of how to do it and all you have to do is replace the executable ls with the name of whatever code you want. The first thing to do is to get rid of the horribly generic JobProfile name and replace it with something sensible that describes the task that will be performed, such as List Directory Contents.

In the parameters block, add a -l so that the input file will be listed in "long form" when we launch the job. (If you are on Windows, leave the parameters blank.) The default installation directory is correct for this command, so the only thing left to do is generate the launcher. There are some options available that bear mentioning for the executable field. You can use some ${installDir} as a prefix for your executable name if it is not installed on your path and ${inputFile} as a place holder for the name of the input file. Job Launchers append the name of the input file by default, but if you include the ${inputFile} flag, it will put the name of the input file exactly in place of that variable. Also add a couple more machines to the Launcher in the Hostnames table. For this exercise, it is enough to add localhost.localdomain and 127.0.0.1 to the list (even though localhost is already there). If you have MPI on your machine, go ahead and check the Enable MPI box, click the Save button and then set the Default Number of Processes that appears in the box. Note a couple of bugs here: Adding something to the table requires clicking into another field, like the name and clicking off again before the save option is enabled. Launching a job on localhost may require a password on some machines. Enabling either OpenMP, TBB or sets of data files do not work either. We are fixing these.

Your profile should now look something like this:

If there were no errors when you clicked the Save button, you should see Ready to Process! beside the Job Profile Editor name at the top of the form. Click the Process drop-down box, select Create a Job Launcher and click Go! If everything was OK in your configuration, Done! will appear in place of Ready to Process!

Launching your new Job Launcher

The new Job Launcher that you created above is ready to be loaded into NiCE and used. You have to restart NiCE for this to work. The Job Profile that you created will appear in the NiCE Item Viewer and you may modify it as you wish, but keep in mind that each time you change it you will have to restart NiCE to load those changes.

Note: There is currently no good way to do refresh in NiCE, so for now we just have to live with a full restart.

Create a new launcher based on your Job Launcher by clicking the Create an Item button and selecting List Directory Contents from the list. A new instance of the List Directory Contents launcher should appear. Feel free to set change some options, such as the input file, number of cores or the target hostname. If you change any options, be sure to save!

Note that there might be some minor differences in the format of your window, as we are still modifying the user interface of this particular piece.

Select Launch the Job from the Process drop-down box and hit Go. You should see the word Done! appear in the status box when the job is finished. If you do not, there was a problem.

You can view all of the output from the job by clicking the the Output Data tab at the bottom of the form and selecting an output file from the list.

Next Steps

The Job Profile Editor is a great way to get a job launching in NiCE quickly and it works for almost every type of scientific simulation or script out there. However, there are the rare situations where it is better to have a Java-based plug-in, like when a third-party library needs to be queried before the job can be launched or when input needs to be generated on the fly. It is also much easier to deploy a Java-based plug-in as part of NiCE because it can be packaged with the executable whereas the file written by the Job Profile Editor would need to be distributed to each user manually.

For more information on how to build a Java plug-in, continue to the next part: Your First Plug-in Part 3: Creating a Java-based Job Launcher with an OSGi NiCE plug-in.