Recent changes to Home

Home modified by John Arcoman

John Arcoman — Wed, 19 Feb 2014 00:59:49 -0000

--- v3
+++ v4
@@ -1,108 +1,21 @@
-The following major packages will be released in the coming weeks as pre-compiled packages with source code.
+# Introduction

-# Crawler and Analysis Packages
+The aim of the ARCOMEM project is the development of methods and tools for the
+implementation of a socially aware and semantic driven Web preservation
+model.

-## Guided crawling: Adapted Heritrix plus Online Analysis
+The crawling and archiving of web content is still a relatively new field of
+research and many of the tools which currently perform these tasks are
+relatively simple, crawling the web without regard to the actual content of the
+web. Throughout the project a large number of components have been developed to
+collect content from the Web and Social Web while focusing on the user-specified
+topics, to analyse it from semantic and social perspectives and to enable Web
+archive access by different facets.

-This package includes an adapted version of the Heritrix crawler that allows a
-prioritisation of the crawler queue as well as a basic online analysis of
-crawled content. The crawler takes an intelligent crawl specification (ICS)
-created by hand or, optionally, with the crawler cockpit by the Crawler Cockpit
-(s. applications below) and uses the results of the online analysis for crawler
-guidance. The guidance is based on keywords and entities as specified in the
-ICS. Identified links on pages are ranked wrt. coverage of page compared to the
-crawl specification. The results of the crawl are focused Web archives.
+This documentation includes:

-This package contains in addition the ARCOMEM API crawler for crawling social
-media sites like Twitter, YouTube or Facebook. Links mentioned in social media
-documents, tweets, etc. are extracted and analysed and used as an initial
-seed list for the adaptive Heritrix crawler as described above.
-
-## Offline Analysis
-
-The Offline Analysis package performs various kinds of analysis on Web archive
-content. The results of the analysis are collected in the H2RDF store - a RDF
-store based on HBase - and can either be directly accessed by applications or
-exported as serialised RDF for archiving or further usage. The following
-analysis modules are provided:
-
-* **Semantic Analysis**: Named entities recognition, Event detection, Opinion
-  mining (all based on GATE),Topic detection, Linked Data Enrichments and
-  Consolidation.
-* **Social Web Analysis**: Cultural analysis of Twitter, News comment and tweet
-  diversification, social search, trending topics detection, Twitter dynamics.
-* **Multimedia Analysis**: Face detection from images and videos, duplicate
-  video detection, opinion identification from images.
-
-The packages described above have been developed for medium and large scale
-crawls and content collections in mind. Therefore the ARCOMEM system follows
-the Map Reduce paradigm in order easily distribute the processing among hosts
-within a computing cluster. Apache Hadoop is used as the Map Reduce framework
-complemented with HBase for content handling. H2RDF is a RDF triple store
-developed on top of HBase that allows the storage and handling of large amounts
-of extracted meta information.
-
-# Applications
-
-Within ARCOMEM two major applications have been developed for crawler handling
-and Web archive access:
-
-## Crawler Cockpit
-
-The crawler cockpit offers integrated features managed through a web
-interface. It manages the main part of the Web Archiving process: creating and
-launching campaigns, and viewing statistics about the crawls. A campaign is
-described by an intelligent crawl definition, which associates content target
-to crawl parameters (schedule and technical parameters). The content definition
-is made of: distinct named entities (e.g. person, place, and organisation),
-time period, free keywords, social media categories, etc.. At the end of the
-crawls, users get access to an overview of the data collected through different
-widgets.
-
-## SARA – Search and Retrieval Application
-
-The Search and Retrieval Application (SARA) is a web application that provides
-an intuitive user interface for search and retrieval of archived web
-documents. It enables users to full-text search and semantic queries to an
-incredibly fast indexed archive. The raw content and the semantic metadata are
-indexed in Solr. Free text search as well as query enabled semantic search
-comprised the search functionality. The returned results are web resources that
-match the search string. The results can be further refined various facets like
-topics, entities, opinions, etc.
-
-# ARCOMEM Light Tools
-
-Since the major tools released by ARCOMEM are based on Apache Hadoop and HBase
-require therefore installation ad handling of many additional libraries the
-ARCOMEM consortium decided to release ``light'' version of certain tools. This
-will allow users to easily try some ARCOMEM analysis tools without the burden
-of complex system handling.
-
-## ARCOMEM Lightweight Semantic Analysis (ARCOLight)
-
-ARCOMEM Lightweight Semantic Analysis (ARCOLight) is an autonomous system for
-the extraction of semantic information and their linked data enrichment. It
-consists of two main components, which manage the extraction of named entities
-from archived websites and the enrichment of these entities with semantic
-information. ARCOLight does not depend on any external systems. This allows
-users to easily install and run it on existing Web archives. ARCOLight takes a
-collection of WARC files as input and produces RDF with extracted information
-as output. This allows the users to integrate the generated information into
-their systems by using existing tools. The incorporation of knowledge bases
-during the enrichment process can either be performed online or offline, after
-downloading a dump of the needed data. This makes ARCOLight a flexible,
-lightweight as well as portable system. It is easy to deploy and works without
-any technical prerequisites.
-
-## ARCOMEM Named Entity Evolution Recognizer (NEER)
-
-NEER is an unsupervised method for named entity evolution recognition
-independent of external knowledge sources. It finds time periods with high
-likelihood of evolution. By analysing only these time periods using a sliding
-window co-occurrence method it captures evolving terms in the same
-context. Thus it avoids comparing terms from widely different periods in time
-and overcome a severe limitation of existing methods for named entity
-evolution.
-
-[[members limit=20]]
-[[download_button]]
+* [instructions](TryIt) to start using parts of, or the whole ARCOMEM
+  system quickly using pre-compiled code;
+* the description of the system's [architecture](Architecture);
+* a [developer guide](DevGuide) for people wishing to extend the ARCOMEM
+  system.

Home modified by John Arcoman

John Arcoman — Mon, 13 Jan 2014 10:51:16 -0000

--- v2
+++ v3
@@ -1,5 +1,4 @@
-The following major packages
-will be released in the coming weeks as pre-compiled packages with source code.
+The following major packages will be released in the coming weeks as pre-compiled packages with source code.

 # Crawler and Analysis Packages

Home modified by John Arcoman

John Arcoman — Mon, 13 Jan 2014 10:50:38 -0000

--- v1
+++ v2
@@ -1,8 +1,109 @@
-Welcome to your wiki!
+The following major packages
+will be released in the coming weeks as pre-compiled packages with source code.

-This is the default page, edit it as you see fit. To add a new page simply reference it within brackets, e.g.: [SamplePage].
+# Crawler and Analysis Packages

-The wiki uses [Markdown](/p/arcomem/wiki/markdown_syntax/) syntax.
+## Guided crawling: Adapted Heritrix plus Online Analysis
+
+This package includes an adapted version of the Heritrix crawler that allows a
+prioritisation of the crawler queue as well as a basic online analysis of
+crawled content. The crawler takes an intelligent crawl specification (ICS)
+created by hand or, optionally, with the crawler cockpit by the Crawler Cockpit
+(s. applications below) and uses the results of the online analysis for crawler
+guidance. The guidance is based on keywords and entities as specified in the
+ICS. Identified links on pages are ranked wrt. coverage of page compared to the
+crawl specification. The results of the crawl are focused Web archives.
+
+This package contains in addition the ARCOMEM API crawler for crawling social
+media sites like Twitter, YouTube or Facebook. Links mentioned in social media
+documents, tweets, etc. are extracted and analysed and used as an initial
+seed list for the adaptive Heritrix crawler as described above.
+
+## Offline Analysis
+
+The Offline Analysis package performs various kinds of analysis on Web archive
+content. The results of the analysis are collected in the H2RDF store - a RDF
+store based on HBase - and can either be directly accessed by applications or
+exported as serialised RDF for archiving or further usage. The following
+analysis modules are provided:
+
+* **Semantic Analysis**: Named entities recognition, Event detection, Opinion
+  mining (all based on GATE),Topic detection, Linked Data Enrichments and
+  Consolidation.
+* **Social Web Analysis**: Cultural analysis of Twitter, News comment and tweet
+  diversification, social search, trending topics detection, Twitter dynamics.
+* **Multimedia Analysis**: Face detection from images and videos, duplicate
+  video detection, opinion identification from images.
+
+The packages described above have been developed for medium and large scale
+crawls and content collections in mind. Therefore the ARCOMEM system follows
+the Map Reduce paradigm in order easily distribute the processing among hosts
+within a computing cluster. Apache Hadoop is used as the Map Reduce framework
+complemented with HBase for content handling. H2RDF is a RDF triple store
+developed on top of HBase that allows the storage and handling of large amounts
+of extracted meta information.
+
+# Applications
+
+Within ARCOMEM two major applications have been developed for crawler handling
+and Web archive access:
+
+## Crawler Cockpit
+
+The crawler cockpit offers integrated features managed through a web
+interface. It manages the main part of the Web Archiving process: creating and
+launching campaigns, and viewing statistics about the crawls. A campaign is
+described by an intelligent crawl definition, which associates content target
+to crawl parameters (schedule and technical parameters). The content definition
+is made of: distinct named entities (e.g. person, place, and organisation),
+time period, free keywords, social media categories, etc.. At the end of the
+crawls, users get access to an overview of the data collected through different
+widgets.
+
+## SARA – Search and Retrieval Application
+
+The Search and Retrieval Application (SARA) is a web application that provides
+an intuitive user interface for search and retrieval of archived web
+documents. It enables users to full-text search and semantic queries to an
+incredibly fast indexed archive. The raw content and the semantic metadata are
+indexed in Solr. Free text search as well as query enabled semantic search
+comprised the search functionality. The returned results are web resources that
+match the search string. The results can be further refined various facets like
+topics, entities, opinions, etc.
+
+# ARCOMEM Light Tools
+
+Since the major tools released by ARCOMEM are based on Apache Hadoop and HBase
+require therefore installation ad handling of many additional libraries the
+ARCOMEM consortium decided to release ``light'' version of certain tools. This
+will allow users to easily try some ARCOMEM analysis tools without the burden
+of complex system handling.
+
+## ARCOMEM Lightweight Semantic Analysis (ARCOLight)
+
+ARCOMEM Lightweight Semantic Analysis (ARCOLight) is an autonomous system for
+the extraction of semantic information and their linked data enrichment. It
+consists of two main components, which manage the extraction of named entities
+from archived websites and the enrichment of these entities with semantic
+information. ARCOLight does not depend on any external systems. This allows
+users to easily install and run it on existing Web archives. ARCOLight takes a
+collection of WARC files as input and produces RDF with extracted information
+as output. This allows the users to integrate the generated information into
+their systems by using existing tools. The incorporation of knowledge bases
+during the enrichment process can either be performed online or offline, after
+downloading a dump of the needed data. This makes ARCOLight a flexible,
+lightweight as well as portable system. It is easy to deploy and works without
+any technical prerequisites.
+
+## ARCOMEM Named Entity Evolution Recognizer (NEER)
+
+NEER is an unsupervised method for named entity evolution recognition
+independent of external knowledge sources. It finds time periods with high
+likelihood of evolution. By analysing only these time periods using a sliding
+window co-occurrence method it captures evolving terms in the same
+context. Thus it avoids comparing terms from widely different periods in time
+and overcome a severe limitation of existing methods for named entity
+evolution.

 [[members limit=20]]
 [[download_button]]

Home modified by John Arcoman

John Arcoman — Wed, 08 Jan 2014 14:02:24 -0000

Welcome to your wiki!

This is the default page, edit it as you see fit. To add a new page simply reference it within brackets, e.g.: [SamplePage].

The wiki uses Markdown syntax.

Project Members:

John Arcoman (admin)