Recent changes to TryIt

TryIt modified by John Arcoman

John Arcoman — Wed, 26 Feb 2014 23:50:49 -0000

--- v5
+++ v6
@@ -7,7 +7,7 @@
 with source code.

 The ARCOMEM system was tested under Debian Squeeze, with Cloudera's Hadoop and
-HBase CDH 4.4.0, Oracle's Java 1.7 JDK and python 2.6.
+HBase CDH 4.5.0, Oracle's Java 1.7 JDK and python 2.6.

 # Crawler and Analysis Packages

@@ -32,13 +32,28 @@
 * [Hadoop and HBase](HadoopHBase);
 * the [Triple Store](KB) (our RDF store based on HBase, here used to serve
   the [ICS](CrawlSpec));
-* [setting up the ICS by hand](KB.html#Python_interface) or using the
-  [Crawler cockpit](Cockpit);
-* the [Online analysis](OnlineQuickStart);
+* [setting up the ICS by hand](KB.html#Python_interface) (requires the
+  `rdfstore` sources) or using the [Crawler cockpit](Cockpit) (requires
+  the `cockpit` sources);
 * [Adaptive Heritrix](AdaptiveHeritrix);
-* and optionally, the [API crawler](APICrawler) to get many relevant seeds
+* optionally, the [API crawler](APICrawler) to get many relevant seeds
   from social sites quickly; note that many offline modules also work
-  specifically on social sites content.
+  specifically on social sites content;
+* the [Online analysis](OnlineQuickStart).
+
+**Quick start**: get the JDK and `hadoop_hbase_setup.sh`, put them under
+root's home and follow the set-up instructions of the first component above.
+
+Put in `hbase`'s home `zookeeper-3.4.3_light.tar.gz`, `ApiCalls.tgz` and
+`H2RDF.jar`, and follow the set-up instructions of the second component.
+
+You will also need the latest `rdfstore` or `cockpit` sources.
+
+Install the adaptive Heritrix and install the cron job to move the WARCs.
+
+Lastly, put in `hbase`'s home `arcomem-framework.online.tgz`, and follow the
+installation instructions (section "Analysis side", no configuration
+modification is needed if you are running everything locally).

 ## Offline Analysis

@@ -65,12 +80,12 @@

 ### Set-up

-Quick start: get the JDK and `hadoop_hbase_setup.sh`, put them under root's
-home and follow the set-up instructions of the first component below. Create an
-arcomem user and put in his home `zookeeper-3.4.3_light.tar.gz` and
-`H2RDF.jar`, and follow the set-up instructions of the second
-component. Lastly, put in hbase's home `sample_af_export.warc.gz` and
-`arcomem-framework.offline.tgz`, and run the commands below.
+**Quick start**: get the JDK and `hadoop_hbase_setup.sh`, put them under root's
+home and follow the set-up instructions of the first component below.  Put in
+`hbase`\s home `zookeeper-3.4.3_light.tar.gz`, `H2RDF.jar` and `ApiCalls.tgz`,
+and follow the set-up instructions of the second component. Lastly, put in
+hbase's home `sample_af_export.warc.gz` and `arcomem-framework.offline.tgz`,
+and run the commands below.

 Components:

@@ -135,19 +150,19 @@

 ## ARCOMEM Lightweight Semantic Analysis (ARCOLight)

-ARCOMEM Lightweight Semantic Analysis (ARCOLight) is an autonomous system for
-the extraction of semantic information and their linked data enrichment. It
-consists of two main components, which manage the extraction of named entities
-from archived websites and the enrichment of these entities with semantic
-information. ARCOLight does not depend on any external systems. This allows
-users to easily install and run it on existing Web archives. ARCOLight takes a
-collection of WARC files as input and produces RDF with extracted information
-as output. This allows the users to integrate the generated information into
-their systems by using existing tools. The incorporation of knowledge bases
-during the enrichment process can either be performed online or offline, after
-downloading a dump of the needed data. This makes ARCOLight a flexible,
-lightweight as well as portable system. It is easy to deploy and works without
-any technical prerequisites.
+ARCOMEM Lightweight Semantic Analysis (ARCOLight) is an autonomous system
+for the extraction of semantic information and their linked data enrichment.
+It consists of two main components, which manage the extraction of named
+entities from archived websites and the enrichment of these entities with
+semantic information. ARCOLight does not depend on any external systems.
+This allows users to easily install and run it on existing Web archives.
+ARCOLight takes a collection of WARC files as input and produces an output
+with extracted information in XML as well as RDF/Turtle format. This allows
+the users to integrate the generated information into their systems by using
+existing tools. ArcoLight incorporates online knowledge bases during the
+enrichment process and does not depend on any offline versions to be
+available. This makes ARCOLight a flexible, lightweight as well as portable
+system. It is easy to deploy and works without any technical prerequisites.

 ## ARCOMEM Named Entity Evolution Recognizer (NEER)

TryIt modified by John Arcoman

John Arcoman — Wed, 26 Feb 2014 11:09:39 -0000

TryIt modified by John Arcoman

John Arcoman — Thu, 20 Feb 2014 10:34:06 -0000

--- v3
+++ v4
@@ -70,7 +70,7 @@
 arcomem user and put in his home `zookeeper-3.4.3_light.tar.gz` and
 `H2RDF.jar`, and follow the set-up instructions of the second
 component. Lastly, put in hbase's home `sample_af_export.warc.gz` and
-`arcomem-framework.tgz`, and run the commands below.
+`arcomem-framework.offline.tgz`, and run the commands below.

 Components:

TryIt modified by John Arcoman

John Arcoman — Thu, 20 Feb 2014 00:17:26 -0000

--- v2
+++ v3
@@ -66,10 +66,11 @@
 ### Set-up

 Quick start: get the JDK and `hadoop_hbase_setup.sh`, put them under root's
-home and follow the set-up instructions of the first item. Then, create an
-arcomem user, put in his home `zookeeper-3.4.3_light.tar.gz` and `H2RDF.jar`,
-and follow the set-up instructions of the second item. Lastly, put in arcomem's
-home `arcomem-framework.tgz`, and `sample.warc.gz`, and run the commands below.
+home and follow the set-up instructions of the first component below. Create an
+arcomem user and put in his home `zookeeper-3.4.3_light.tar.gz` and
+`H2RDF.jar`, and follow the set-up instructions of the second
+component. Lastly, put in hbase's home `sample_af_export.warc.gz` and
+`arcomem-framework.tgz`, and run the commands below.

 Components:

@@ -77,20 +78,19 @@
 * the [Triple Store](KB);
 * the [offline framework](OfflineAnalysis).

-Offline quick start steps:
+Offline quick start steps, as user hbase:

-    tar xzf arcomem-framework.tgz
+    tar xzf arcomem-framework.offline.tgz
     cd arcomem-framework

     # load some WARC content into HBase
     PATH=$PATH:$PWD/ingestion_scripts/hbase_side
-    hadoop fs -copyFromLocal ~/sample.warc.gz /exchange/import/test/
+    hadoop fs -copyFromLocal ~/sample_af_export.warc.gz /exchange/import/test/
     cd ingestion_scripts/hbase_side
     ./load_warcs test test test bulk_load

     # build an off-line configuration and run it
-    cd ../..
-    cd configurations
+    cd ../../configurations
     ./build_conf.sh test localhost test_kb gate > test_gate.xml
     cd ..
     tools/run_combined_off-line configurations/test_gate.xml

TryIt modified by John Arcoman

John Arcoman — Wed, 19 Feb 2014 01:11:16 -0000

--- v1
+++ v2
@@ -6,8 +6,8 @@
 major packages will be released in the coming weeks as pre-compiled packages
 with source code.

-The ARCOMEM system was tested under Debian Squeeze, with Cloudera\'s Hadoop and
-HBase CDH 4.4.0, Oracle\'s Java 1.7 JDK and python 2.6.
+The ARCOMEM system was tested under Debian Squeeze, with Cloudera's Hadoop and
+HBase CDH 4.4.0, Oracle's Java 1.7 JDK and python 2.6.

 # Crawler and Analysis Packages

@@ -129,7 +129,7 @@

 Since the major tools released by ARCOMEM are based on Apache Hadoop and HBase
 require therefore installation ad handling of many additional libraries the
-ARCOMEM consortium decided to release \"light\" version of certain tools. This
+ARCOMEM consortium decided to release "light" version of certain tools. This
 will allow users to easily try some ARCOMEM analysis tools without the burden
 of complex system handling.

TryIt modified by John Arcoman

John Arcoman — Wed, 19 Feb 2014 01:00:29 -0000

The whole system based on the Heritrix crawler is released as open source to
the public. Since many components or composite tools are of interest also for
other areas and usage scenarios, the ARCOMEM consortium defined a number of
pre-packaged tools which can be used independently from each other. By
combining all packages the full ARCOMEM system can be built. The following
major packages will be released in the coming weeks as pre-compiled packages
with source code.

The ARCOMEM system was tested under Debian Squeeze, with Cloudera\'s Hadoop and
HBase CDH 4.4.0, Oracle\'s Java 1.7 JDK and python 2.6.

Crawler and Analysis Packages

Guided crawling: Adapted Heritrix plus Online Analysis

This package includes an adapted version of the Heritrix crawler that allows a
prioritisation of the crawler queue as well as a basic online analysis of
crawled content. The crawler takes an intelligent crawl specification (ICS)
created by hand or, optionally, with the crawler cockpit by the Crawler Cockpit
(s. applications below) and uses the results of the online analysis for crawler
guidance. The guidance is based on keywords and entities as specified in the
ICS. Identified links on pages are ranked wrt. coverage of page compared to the
crawl specification. The results of the crawl are focused Web archives.

This package contains in addition the ARCOMEM API crawler for crawling social
media sites like Twitter, YouTube or Facebook. Links mentioned in social media
documents, tweets, etc. are extracted and analysed and used as an initial
seed list for the adaptive Heritrix crawler as described above.

Set-up

Hadoop and HBase;
the Triple Store (our RDF store based on HBase, here used to serve
the ICS);
setting up the ICS by hand or using the
Crawler cockpit;
the Online analysis;
Adaptive Heritrix;
and optionally, the API crawler to get many relevant seeds
from social sites quickly; note that many offline modules also work
specifically on social sites content.

Offline Analysis

The Offline Analysis package performs various kinds of analysis on Web archive
content. The results of the analysis are collected in the H2RDF store and can
either be directly accessed by applications or exported as serialised RDF for
archiving or further usage. The following analysis modules are provided:

Semantic Analysis: Named entities recognition, Event detection, Opinion
mining (all based on GATE), Topic detection, Linked Data Enrichments and
Consolidation.
Social Web Analysis: Cultural analysis of Twitter, News comment and tweet
diversification, social search, trending topics detection, Twitter dynamics.
Multimedia Analysis: Face detection from images and videos, duplicate
video detection, opinion identification from images.

The packages described above have been developed for medium and large scale
crawls and content collections in mind. Therefore the ARCOMEM system follows
the Map Reduce paradigm in order easily distribute the processing among hosts
within a computing cluster. Apache Hadoop is used as the Map Reduce framework
complemented with HBase for content handling. H2RDF is a RDF triple store
developed on top of HBase that allows the storage and handling of large amounts
of extracted meta information.

Set-up

Quick start: get the JDK and hadoop_hbase_setup.sh, put them under root's
home and follow the set-up instructions of the first item. Then, create an
arcomem user, put in his home zookeeper-3.4.3_light.tar.gz and H2RDF.jar,
and follow the set-up instructions of the second item. Lastly, put in arcomem's
home arcomem-framework.tgz, and sample.warc.gz, and run the commands below.

Components:

Offline quick start steps:

tar xzf arcomem-framework.tgz
cd arcomem-framework

# load some WARC content into HBase
PATH=$PATH:$PWD/ingestion_scripts/hbase_side
hadoop fs -copyFromLocal ~/sample.warc.gz /exchange/import/test/
cd ingestion_scripts/hbase_side
./load_warcs test test test bulk_load

# build an off-line configuration and run it
cd ../..
cd configurations
./build_conf.sh test localhost test_kb gate > test_gate.xml
cd ..
tools/run_combined_off-line configurations/test_gate.xml

You can follow the map-reduce job at http://localhost:50030/.

Applications

Within ARCOMEM two major applications have been developed for crawler handling
and Web archive access:

Crawler Cockpit

The crawler cockpit offers integrated features managed through a web
interface. It manages the main part of the Web Archiving process: creating and
launching campaigns, and viewing statistics about the crawls. A campaign is
described by an intelligent crawl definition, which associates content target
to crawl parameters (schedule and technical parameters). The content definition
is made of: distinct named entities (e.g. person, place, and organisation),
time period, free keywords, social media categories, etc.. At the end of the
crawls, users get access to an overview of the data collected through different
widgets.

SARA – Search and Retrieval Application

The Search and Retrieval Application (SARA) is a web application that provides
an intuitive user interface for search and retrieval of archived web
documents. It enables users to full-text search and semantic queries to an
incredibly fast indexed archive. The raw content and the semantic metadata are
indexed in Solr. Free text search as well as query enabled semantic search
comprised the search functionality. The returned results are web resources that
match the search string. The results can be further refined various facets like
topics, entities, opinions, etc.

ARCOMEM Light Tools

Since the major tools released by ARCOMEM are based on Apache Hadoop and HBase
require therefore installation ad handling of many additional libraries the
ARCOMEM consortium decided to release \"light\" version of certain tools. This
will allow users to easily try some ARCOMEM analysis tools without the burden
of complex system handling.

ARCOMEM Lightweight Semantic Analysis (ARCOLight)

ARCOMEM Lightweight Semantic Analysis (ARCOLight) is an autonomous system for
the extraction of semantic information and their linked data enrichment. It
consists of two main components, which manage the extraction of named entities
from archived websites and the enrichment of these entities with semantic
information. ARCOLight does not depend on any external systems. This allows
users to easily install and run it on existing Web archives. ARCOLight takes a
collection of WARC files as input and produces RDF with extracted information
as output. This allows the users to integrate the generated information into
their systems by using existing tools. The incorporation of knowledge bases
during the enrichment process can either be performed online or offline, after
downloading a dump of the needed data. This makes ARCOLight a flexible,
lightweight as well as portable system. It is easy to deploy and works without
any technical prerequisites.

ARCOMEM Named Entity Evolution Recognizer (NEER)

NEER is an unsupervised method for named entity evolution recognition
independent of external knowledge sources. It finds time periods with high
likelihood of evolution. By analysing only these time periods using a sliding
window co-occurrence method it captures evolving terms in the same
context. Thus it avoids comparing terms from widely different periods in time
and overcome a severe limitation of existing methods for named entity
evolution.