OfflineOutputs modified by John Arcoman

John Arcoman — Wed, 19 Feb 2014 00:34:14 -0000

Outputting Data from Offline Modules

Every offline processing module has the option of how it outputs its data.
It can output to a Hadoop sequence file, write some data back to the HBase or
it can output semantic information to a triple-store.

Note that offline modules that work on sequence files, local files or the
triple-store are just basic Hadoop MapReduce jobs so extend the MapReduceSpecGeneric
class that is part of the Hadoop framework.

Writing Output to a Sequence File

If a module writes large amounts of data or writes binary data that would not sensibly
need to be kept in the HBase, the module can use the Hadoop file system to write data
to a sequence file. Sequence files are available over the cluster, so the data that
is written to them can be used other tasks which might occur on other nodes.

To write the output of a module to a sequence file, the module should extend the
OfflineProcess
class as normal. However, the final generic type (the OUTPUT_FORMAT)
should be of the type:

SequenceFileOutput<KEY_TYPE,VALUE_TYPE>

A sequence file, just like an HBase, is a key-value store, so the types of the key and
value need to be supplied. In most cases the KEY_TYPE will be the same key type as the
HBase (you will key on the URI of the web object), which is Text. If you\'re writing
binary you can use BytesWritable as the value type (just a byte array of data).

If the type is map-heavy, then the reducer needs to be the default reducer (rather than
the the NullReducer).
This is because the sequence file is defined as the output of the
whole job (map and reduce), so the reducer needs to output the sequence file (the default
one will do this for you if you define your module to output a sequence file).

The following shows a skeleton for a class that will map over the elements of the HBase
and then output a sequence file (with no reducer):

public class SequenceFileWriterModule
    extends OfflineProcess<
        Text, BytesWritable,
        Text, BytesWritable,
        SequenceFileOutputFormat<Text, BytesWritable>
    >
{
    @Override
    public Class<? extends OfflineProcessMapper<Text, BytesWritable>>
        getMapperClass() throws ClassNotFoundException
    {
            return MyMapperClass.class;
    }

    @SuppressWarnings("unchecked")
    @Override
    public Class<? extends Reducer<Text, BytesWritable, Text, BytesWritable>>
        getReducerClass() throws ClassNotFoundException
    {
        // we don't actually use the reducer, so the default one is returned
        return (Class<? extends Reducer<Text, BytesWritable, Text, BytesWritable>>)
            (Class<?>) Reducer.class;
    }

    @Override
    public void setupJob(Job job)
    {
        super.setupJob(job);
        job.setNumReduceTasks(0);
    }
}

Note that we return a default reducer for the reducer class (rather than a
NullReducer)
[The double cast there is a fix for a generic bug in certain versions of Java].

Writing Output back the to HBase

There may be modules which wish to store information into the HBase for better controlling the
input to subsequent tasks.

The interface to rows of the HBase is provided by an API called
AMResourceVersion.
This class provides methods for getting and setting values of a specific row in the HBase. If you want to store
information to this row, you need to ensure that an appropriate cell is available into which to store it.
To do this, you simply need to provide a name for the cell. The cell must be put within the \'Content\'
column family. The usual way is to extend the
AMResourceVersion
with your own methods for writing your data.

public class MyResourceVersion extends AMResourceVersion
{
    public String getMyData()
    {
        return (String) resource.getQualifierValue(
            ResourceVersion.CONTENT_CF,     // Column family must be this
            "my.data.cell"
        );
    }

    public void setGateSimpleXml( String myData )
    {
        resource.setQualifierValue(
            ResourceVersion.CONTENT_CF,     // Column family must be this
            "my.data.cell", myData );
    }
}

Then, during the mapping or reducing, you can simply retrieve a row into your class
from the context then use your class to set and/or get values.

MyResourceVersion resource = wrapper.getResource().getLatestRowVersion( MyResourceVersion.class );
resource.setMyData( "myData" );
String myData = resource.getMyData();

Note: If the data is to be consistently used by many modules, it may be an idea to update the
AMResourceVersion
class rather than extending it as has been done for GateXml data.

Writing to the Triple Store

The ARCOMEM framework provides a simple method for constructing a
TripleStoreConnector
object which will give access to a knowledgebase which can be queried or written to.
If your offline process requires this form of output, then in the
setup(Job) method, the offline process can call

TripleStoreConnector tsc = TripleStoreConnector.newConnector( context.getConfig() )

The
newConnector(config)
method takes a configuration object from which
it can determine which triple store connector implementation to invoke.
When running as a Hadoop Map-Reduce job, the configuration can be retrieved
from the task’s context as above.
This TripleStoreConnector
object has a
writeRDF(String)
and
writeTriple(Statement)
methods which provide two separate means for writing data to the triple store.
The first allows the writing of a batch of data encoded in RDF while the second
allows single triples to be asserted into the knowledgebase. These methods
provide no form of validation to the data that any code passes to it
(unless the underlying triple store does this). So one way of providing a set
of triples that conform to the ARCOMEM ontology is to use the data-model
Java classes which are available in a separate subproject within the ARCOMEM
codebase (see the Data Model section for information).

Recent changes to OfflineOutputs

OfflineOutputs modified by John Arcoman

Outputting Data from Offline Modules

Writing Output to a Sequence File

Writing Output back the to HBase

Writing to the Triple Store