RAIDmap Wiki

An application to help researchers organize their research data.

Status: Pre-Alpha

Brought to you by: alexball

DeveloperGuide

Introduction
Setting up a development environment for RAIDmap
- Java
- Maven
- Eclipse
Working with the source code snapshot
Working with the SourceForge Git repository
- Working from the Command Prompt
- Working within Eclipse
Building the Windows installer with IzPack
RAIDmap architecture
Development tasks

Introduction

RAIDmap (for Research Activity Information Development Mapping) is a prototype application to help researchers organize their research data records. It provides a file metadata harvester and a means of visualizing the associations between both digital and physical records which constitute research data sets.

This Developer Guide was originally written as a deliverable (Document redm5rep120510ab10) of the REDm-MED Project funded by JISC as part of the second phase of the Managing Research Data Programme.

Setting up a development environment for RAIDmap

Java

Make sure that Java is installed. You will need the Standard Edition of the Java Development Kit version 5 or later. The latest version may be downloaded from the Oracle Web site here: http://www.oracle.com/technetwork/java/javase/downloads/
Set the Environment Variable JAVA_HOME to the directory in which the Java JDK has been installed, e.g. C:\Program Files\Java\jdk1.5.0_12. You can do this in Windows by
- holding down the Windows key and pressing Pause, or
- running start sysdm.cpl from the Command Prompt, or
- opening the Start menu, right-clicking on Computer and
  selecting Properties
then selecting the Advanced tab (via Advanced System Settings in some versions of Windows) and clicking the Environment Variables button. Note that you do not need to wrap the pathname in quote marks even if it contains spaces.
You should also ensure that %JAVA_HOME%\bin (Windows) or $JAVA_HOME/bin (otherwise) appears in your Path Environment Variable.

Maven

Make sure that Maven is installed. You will need version 2.2 or later. The latest version of Maven is available for download from http://maven.apache.org/download.html
Set the Environment Variable M2_HOME to the directory in which Maven has been installed, e.g. C:\Program Files\Apache Software Foundation\apache-maven-3.0.4. For how to set Environment Variables, see above.
Similarly, set the Environment Variable M2 to %M2_HOME%\bin (Windows) or $M2_HOME/bin (otherwise).
Ensure that your Path Environment Variable contains %M2% (Windows) or $M2 (otherwise).
Verify that your installation is working by running mvn –version from the Command Prompt. You should get a message similar to that shown below.

Testing to ensure Maven is correctly installed

Eclipse

Eclipse is an integrated development environment written in and primarily used for coding Java, though support for other languages is available. Some support is given later in this guide for working with the RAIDmap code in Eclipse. If you do not have a working installation of Eclipse but would like to, several different configurations of the software may be obtained from http://www.eclipse.org/downloads/; the ‘Eclipse IDE for Java Developers’ is recommended as a suitable initial configuration.

Please ensure the Maven Integration for Eclipse (m2e) plugin has been installed. This can be done within Eclipse either through the Eclipse Marketplace (Help > Eclipse Marketplace) or by selecting Help then Install New Software, and searching the main software repository (http://download.eclipse.org/releases/release name) for ‘Maven’. Once you have this installed, it may take Eclipse several hours to download and install the index for the official central Maven repository. It is recommended, therefore, that once this is done you open Window > Preferences, navigate to Maven and untick the option to Download repository index updates on startup. Within the same preferences section, under Installations you should also add and select your regular Maven install instead of the embedded version, and either there or under User Settings ensure that Eclipse has found your local Maven repository: by default, this is in a (possibly hidden) directory .m2 in your home or user directory.

Working with the source code snapshot

There are two ways of obtaining the RAIDmap source code. The first, described in this section, involves the source code snapshot, which is the code underlying the installer files. The second, described in , involves working directly with the very latest version of the code.

Obtaining the RAIDmap source code

The source code snapshot may be downloaded from SourceForge: https://sourceforge.net/projects/raidmap/files/. Once you have obtained your copy of RAIDMap Project Workspace.zip extract it somewhere on your local system.

The RAIDmap application uses some libraries that Maven does not know about by default. Some DOS batch scripts are provided for installing these libraries to your local Maven repository. These scripts may be found within the RAIDMap Project Workspace directory in the following locations:

compendium\MavenExtraLib\ExtraLibInstallCommand.bat
metadata-extractor\MavenExtraLib\ExtraLibInstallCommand.bat

Similar Bash scripts are also provided for other platforms. If running these from the Command Prompt, be sure to navigate to the respective MavenExtraLib directories first so the relative pathnames in the scripts point to the correct locations.

Building the RAIDmap application

Open a Command Prompt and navigate to the RAIDMap Project Workspace directory.
Run the following command and wait for it to finish:
```
mvn clean install -Dmaven.test.skip=true -Dmaven.compiler.source=1.5 -Dmaven.compiler.target=1.5
```
This will install jar files for the various components of the application to the local Maven repository.
At the Command Prompt, navigate to the compendium directory within the RAIDMap Project Workspace directory.
To compile the executable jar file, run the following command:
```
mvn assembly:single -Dmaven.test.skip=true
```
It will be saved in the target subdirectory with the name compendium-1.0-SNAPSHOT-jar-with-dependencies.jar. Rename it raidmap.jar.
Finally run and test the application by navigating to the target directory and running the following from the Command Prompt:
```
java -jar raidmap.jar
```

It is also possible to build the application from within an integrated development environment such as Eclipse (see below), though it is best to check the process works at the Command Prompt using the above instructions before attempting this. The instructions for building RAIDmap within Eclipse are as follows.

In the Package Explorer, navigate to the compendium project and select pom.xml.
Either right-click to get the context menu, or open the Run menu from the menu bar, and then select Run As > Maven build. The second time you do this and subsequently, Eclipse will build the application using Maven. The first time you do this, however, you will be presented with a dialogue box to configure the build process.
Under Goal, enter assembly:single and tick the box marked Skip tests. Then select Run.
The executable jar file will be saved in the target subdirectory with the name compendium-1.0-SNAPSHOT-jar-with-dependencies.jar. Refresh the compendium project within Eclipse, then locate the file and rename it raidmap.jar.

Importing the RAIDmap application into Eclipse

Locate the RAIDmap Project Workspace directory and inspect the compendium and metadata-extractor subdirectories. In each case, ensure that the directory does NOT contain a file named .classpath or a subdirectory named target or .settings. These will be generated by Eclipse on import, and should not exist beforehand.

Open a Command Prompt, navigate to the RAIDmap Project Workspace directory and run the following four commands in sequence:

mvn clean
mvn compile -Dmaven.compiler.source=1.5 -Dmaven.compiler.target=1.5
mvn eclipse:clean
mvn -DdownloadSources=true eclipse:eclipse

Within Eclipse, select File > Import. In the dialogue box that appears, expand the Maven node, select Existing Maven Projects and then Next.
On the screen that appears, locate the Root Directory line, select Browse and then locate and select the RAIDmap Project Workspace directory, then select OK.
Select all the projects that Eclipse finds, then select Finish.
You should now have compendium and metadata-extractor included in your list of projects in the Package Explorer pane. For each one in turn, right-click and select Build Path > Configure Build Path. Select the Libraries tab and remove metadata.jar from the build path if it appears there, then select OK.
If you experience errors relating to the Maven build path for either project, right click on it in the Package Explorer, select Maven > Update Project Configuration then select OK in the dialogue box.

Working with the SourceForge Git repository

If you wish to contribute to the RAIDmap application, you will need to request SSH access to the RAIDmap Git repository on SourceForge; to do this, visit the profile page of one of the administrators and send them a message through the SourceForge interface. Otherwise, the code is available anonymously over HTTP and using the special Git protocol.

Working from the Command Prompt

You will need to have Git installed and working on your computer. Git is a popular distributed version control system; it is particularly well suited to projects where the developers work independently on different copies (forks) of the code and later share and synchronize (merge) their work. For help on installing and configuring Git, see the step-by-step instructions provided by GitHub: https://help.github.com/articles/set-up-git

Open up the Git Bash prompt.
For read-only access to the RAIDmap code, enter the following command:
```
git clone git://git.code.sf.net/p/raidmap/code RAIDmap
```
or for read-write access, use the following command, replacing USERNAME with your SourceForge user name:
```
git clone ssh://USERNAME@git.code.sf.net/p/raidmap/code RAIDmap
```
The code will be downloaded to your local Git repository into the RAIDmap directory.

There is not room here to describe how to use Git to track the changes you make to the source code – there is a helpful online manual you can consult for that: http://git-scm.com/book/en/ – but to get you started some useful commands follow.

git commit -a -m "Message"

This will save a new version of the code, with all the changes made since the last time you saved a version (the -a flag lets you skip an intermediate staging process). The Message bit should be replaced with a description of what you did, so if you ever need to revert the changes you can find where you made them.

git push origin master

Provided you have set up your SSH access correctly, this will upload your local changes to the RAIDmap Git repository on SourceForge.

Working within Eclipse

Recent versions of Eclipse come bundled with a tool called EGit for interacting with Git repositories.

From the menu bar, select File > Import.
In the dialogue box that appears, select Git > Projects from Git then Next.
Select URI then Next.
For read-only access to the RAIDmap code, in the URI box enter git://git.code.sf.net/p/raidmap/code. For read-write access, enter ssh://git.code.sf.net/p/raidmap/code, then add your SourceForge user name and password in the appropriate places. The rest of the dialogue box should be filled in automatically for you. Select Next.
Select the master branch. Select Next.
Here you can customize where the code will be stored locally (Directory) and the name EGit will use for the SourceForge repository (Remote name). Make any changes you wish, then select Finish.
Once the code has downloaded, import the extra libraries into your local Maven repository using the scripts described above.
Select File > Import again. This time select Maven > Existing Maven Projects then Next.
Locate the Root Directory line, select Browse and select the folder into which you downloaded the source code, then select OK.
Make sure only the compendium and metadata-extractor projects are selected, then select Finish.

When working on the code, you can access the EGit functions by right-clicking on the RAIDmap project in Project Explorer and navigating to Team in the context menu. Use Commit to save a new version of your code in your local repository, and use Remote > Push to push these changes up to the SourceForge repository (if you have SSH access).

Building the Windows installer with IzPack

Make sure that IzPack is installed. The latest version may be downloaded from the IzPack Web site here: http://izpack.org/
Prepare the RAIDMap application using Maven build tool (see ). This will produce a jar file called raidmap.jar alongside other files required to run the application.
Create an empty directory somewhere on your system and call it, for example, RAIDmap Installer.
Add the following files and directories from your built application and the Installer directory in the RAIDmap Project Workspace to the RAIDmap Installer directory:
- exe.ico
- install.xml
- Licence.txt
- raidmap.jar
- Readme.txt
- shortcutSpec.xml
- METADATA_BASE
- Skins
- System
Ensure that none of these files or directories has a read-only flag set, and that there are no permissions issues with modifying them.
Within the RAIDMap Installer directory, create a subdirectory called bin.
Locate IzPack’s native directory. On Windows, this is installed by default to C:\Program Files\IzPack\bin\native. Copy this directory to the bin directory you just created.
Open the install.xml file in a text or XML editor. Towards the end are a couple of native elements that need to be altered depending on the OS and architecture of the machine that will run the installer. For 32-bit Windows, they should read as follows:
```
<native type="izpack" name="ShellLink.dll"/>
<native type="3rdparty" name="COIOSHelper.dll" stage="both">
  <os family="windows"/>
</native>
```
while for 64-bit Windows, they should look like this:
```
<native type="izpack" name="ShellLink_x64.dll"/>
  <native type="3rdparty" name="COIOSHelper_x64.dll" stage="both">
  <os family="windows"/>
</native>
```
Save any changes and close the file.
Your RAIDMap Installer directory should now look like the one shown above. Open a Command Prompt within that directory and run the following command, remembering to change the path to the installation directory of IzPack if it is different, and to replace xxbit with either 32bit or 64bit as appropriate:
```
C:\Program Files\IzPack\bin\compile install.xml -h C:\Program Files\IzPack -o raidmap-install-xxbit-windows.jar
```
This will produce an installer file of the name specified above in the RAIDMap Installer directory, ready for use or distribution.

RAIDmap architecture

The primary part of RAIDmap is contained in the compendium folder. The Java source code is contained in src\main\java. The list below indicates the structure of the code but does not enumerate every class or package individually.

com.compendium

This is the top level package containing the classes for launching the application.
- ProjectCompendium: This is the main file that, for example, Eclipse would need to run in order to test the application.
- Launcher: Attempts to locate and launch an existing installation and if one cannot be found, installs the application.
- SaveOutput: Redirects application messages to a log file.
com.compendium.core

This package contains utility functions and constants used throughout the application. Its subpackages deal with the underlying data model and databases.
- CoreCalendar: Contains functions for working with times and dates.
- CoreUtilities: Contains functions for working with files, sanitizing database inputs and outputs, etc.
- ICoreConstants: Defines constants for, among other things, node and link types.
com.compendium.ui

This package defines the elements and behaviour of the node maps.
com.compendium.ui.plaf

This package defines further aspects of the look and feel of map elements such as lines, links and nodes.
com.compendium.ui.toolbars

This package defines the toolbars used by the application.
com.compendium.ui.toolbars.system

This package controls how the placement and arrangement of the toolbars may be customized.
com.compendium.ui.menus

This package defines the applications menu system.
com.compendium.ui.popups

This package defines context menus for various interface elements.
- UILinkPopupMenu: The context menu for linking arrows.
- UINodePopupMenu: The context menu for nodes, including data record nodes.
com.compendium.ui.dialogs

This package defines (most of) the dialogue boxes used by the application, but not all of their contents.
com.compendium.ui.panels

The package defines informationediting panels that are included as tabs within dialogue boxes.
- UILinkEditPanel: Defines the panel that appears when editing the Contents of a linking arrow.
- UINodeEditPanel: Defines the panel that appears when editing the Contents of a node.
com.compendium.ui.linkgroups

This package defines a data model and dialogue box for editing the classifications used for assigning types to links. The default classification is defined in System\resources\LinkGroups\Default.xml and the active one is stored in System\resources\LinkGroups\Databases
com.compendium.ui.stencils

This package defines a data model and dialogue boxes for editing stencil sets within the application. The RAIDmap stencils are located in System\resources\Stencils\RAIDmap_development_process.
com.compendium.ui.tags

This package defines a data model and dialogue boxes for editing hierarchies of tags within the application.
com.compendium.ui.edits

This package allows edits made within the application to be turned into objects, so they can be entered into an undoredo list.
com.compendium.io

This package and its subpackages define how application data may be imported and exported in various formats.
uk.ac.bath.raidmap.metadata

This package provides the dialogue box that wraps around the HTML form for editing the metadata associated with nodes.
uk.ac.bath.raidmap.metadata.pathwatch

This package provides experimental code for ‘watching’ a directory for changes, and allowing metadata to be entered into the RAIDmap database without the main application running.
com.compendium.meeting

This package and its subpackages provide support for recording meetings in the application. This is not a focus for RAIDmap.

The metadata-extractor folder contains the code for extracting metadata from digital objects on the file system. Code in the nz.govt packages has not been altered to any great extent. There are however some additional packages.

uk.ac.bath.raidmap.meta

Provides a script interface for running the metadata extractor from within RAIDmap
- RaidMapCmdLine This is the class called by UINodeEditPanel when a digital object is associated with a record node.
uk.ac.bath.raidmap.meta.harvester

This package provides alternative versions of the Simple Object Harvester and the Complex Object Harvester for use with RaidMapCmdLine.
uk.ac.bath.raidmap.meta.utils

Contains utilities for working with XML documents.

Development tasks

How to extend file types handled by the system

To add a new file type to RAIDmap’s repertoire of formats from which it can extract metadata, the following files must be in place.

The metadata extractor needs a data adapter that tells it how to read a file in the given format. This should be a class file that extends nz.govt.natlib.adapter.DataAdapter, and conventionally should be called something like nz.govt.natlib.adapter.ext.MyformatAdapter (e.g. the adapter for a Windows Bitmap is called nz.govt.natlib.adapter.bmp.BitmapAdapter).

The class should at least implement the abstract methods defined by DataAdapter, that is, acceptsFile, adapt, getOutputType, getInputType, getName, getDescription, and getVersion. For hints on how to do this, please consult the existing set of adapters. These can be found in the metadata-extractor\src\main\java directory.
Document Type Definition corresponding to the XML generated by the adapt method of the adapter, e.g. bmp.dtd. These are kept in the METADATA_BASE\xml directory within both the compendium and metadata-extractor project directories.
XML output template for transforming the XML generated by the adapt method of the adapter into XML that can be used by the RAIDmap form, e.g. bmp_to_raidmap.xslt. Again, for hints on writing such a template, please consult the existing ones. These are also kept in the METADATA_BASE\xml directory within both the compendium and metadata-extractor project directories.

Note that each format can have a different set of mandatory metadata, depending on its significant properties.

Please consider contributing any data adapters and DTDs you write to the Metadata Extractor project as well as to RAIDmap.

How to customize the metadata collected

There are three different metadata forms presented to users.

The form used for physical objects is METADATA_BASE\harvested\raidmap\physicalObject.xml.
The form used for physical specimens is METADATA_BASE\harvested\raidmap\physicalSpecimen.xml.
The form used for digital objects is automatically generated from this template: METADATA_BASE\harvested\raidmap\RaidmapMetadataForm.xslt. Those parts of the metadata not included directly in the template are derived from the output templates used by the metadata extractor (see above).

Note that these appear within both the compendium and metadata-extractor project directories.

Outstanding issues

The RAIDmap software is currently in an early stage of development, and there are many adjustments that need to be made before it can be released for beta testing. Any contributions towards making those adjustments would be warmly welcomed.

A list of outstanding issues is maintained on the SourceForge site: http://sourceforge.net/p/raidmap/tickets/.

Authors: Alex Ball and Uday Thangarajah

Wiki: UserGuide