Download Latest Version TokenGenerator-2.0.jar (7.5 MB)
Email in envelope

Get an email when there's a new version of MARKOS Project

Home / MARKOS 1.3
Name Modified Size InfoDownloads / Week
Parent folder
sources 2014-07-11
0
binaries 2014-07-11
0
README.md 2014-07-11 22.8 kB
0
MARKOS installation guide.docx 2014-07-11 29.0 kB
0
Totals: 4 Items   51.8 kB 0

MARKOS installation guide

This document is intended to document the installation and integration process to set-up the whole or a part of the MarkOS infrastructure. For each component of the system there is a dedicated section explaining all the steps needed to install, configure and compile the code.

If you do not want to compile the code yourself, skip the instructions for compilation and read directly the sections for configuration, installation and execution, using the object code available from the "binaries" directory (only for the Crawler you have to use the source code from the "sources" directory).

1. Crawler

Requirements

The Crawler should run on any operating system capable of running Python 2.7 and MySQL. It has been tested on Ubuntu and SunOS 5.11. Following instruction have been tested on an Ubuntu Virtual Machine available at the following url:

http://www.turnkeylinux.org/lampstack

Download

Download and uncompress the file "Crawler_1.3-sources.zip" from the "sources" directory.

Configuration

One time setup

cd markos-crawler/api

python setup.py develop

Setup setuptools

cd

wget http://peak.telecommunity.com/dist/ez_setup.py

python ez_setup.py

Install pyramid (waitress is a new dependency)

easy_install pyramid

easy_install waitress

One time setup for doapfiend

cd markos-crawler/lib/doapfiend/doapfiend-0.3.3

python setup.py install

Create database structure

mysql -u root -p

source <<......>>/markos-crawler/database/createCrawlerDB.sql

Create a MySQL user with all permissions on the database created by the above script

Edit configuration on $HOME/crawler/config

make sure all paths exist

      [Fetchers] sf\_file\_path is where the data fetcher will look for sourceforge export

[Fetchers] flossmole_file_path tells where flossmole files will be downloaded; if you enable flossmole please make sure you have at least 100GB on this file system.

[Logging] path is the path to the log file, including the file name

[Logging] path_html is not used but must exist

Write Code Analyser IP address and optional port

Configure the [Fetchers] sections to decide which Forges to crawl and how frequently.

Bear in mind that SourceForge data is fetched from an export which is not included in this release. Flossmole at the moment includes only data coming from Github.

Configure the [Database] according to the above user and database

Change startCrawler permissions

chmod +x startCrawler

Execution

Start the crawler

./startCrawler

If you configure a Python virtualenv please edit startCrawler file so that it points to the right Python version.

The crawler starts fetching data from the forges you have configured and after it completes each batch it will notify the Code Analyser which in turn will get the projects data using a url similar to the following one:

http://:6543/projects?batchId=1

it should return something like

"[{\"doap\": [{\"wiki\": [\"http://abdera.apache.org/wiki\"], \"download_page\": \"http .....

2. Code Analyzer

Requirements

Needed operating system: the Code Analyzer can work on Windows and on any UNIX-based operating system (e.g., Linux, Mac OS X, etc).

Download

You can download directly the object code file "code-analyser-1.3.war" from the "binaries" directory or you can follow the following building process.

Building process

Needed compiler, tools, components or library that must be manually installed:

Apache Tomcat 6.x or higher ( http://tomcat.apache.org/download-60.cgi)

MySQL Server 5.x or higher ( http://dev.mysql.com/downloads/mysql/)

The required libraries are already provided with the CA source code or within the CA WAR.

Download and uncompress the file "markos-code-analyser-1.3-sources.zip" from the "sources" directory.

The Code Analyzer is fully written in Java. Thus, importing it in a Java IDE and ensure that the following libraries are in the project build path:

commons-compress-1.7.jar

commons-io 2.4.jar

gson-2.2.2.jar

mysql-connector-java-3.0.12

org.eclipse.core.contenttype_3.4.200.v20120523-2004.jar

org.eclipse.core.jobs_3.5.300.v20120912-155018.jar

org.eclipse.core.resources_3.8.1.v20121114-124432.jar

org.eclipse.core.runtime_3.8.0.v20120912-155025.jar

org.eclipse.emf.ecore_2.8.3.v20130125-0546.jar

org.eclipse.equinox.common_3.6.100.v20120522-1841.jar

org.eclipse.equinox.preferences_3.5.1.v20121031-182809.jar

org.eclipse.equinox.registry_3.5.200.v20120522-1841.jar

org.eclipse.osgi_3.9.1.v20130814-1242.jar

org.eclipse.osgi.services_3.3.100.v20120522-1822.jar

org.eclipse.osgi.util_3.2.300.v20120913-144807.jar

org.eclipse.text_3.5.200.v20120523-1310.jar

org.eclipse.wst.jsdt.core_1.1.202.v201208171701.jar

Configuration

The following parameters must be set in file web.xml present in webapps/Analyser/WEB-INF:

crawlerIpAndPort: the IP of the machine on which the Crawler component is installed followed by the number of the port on which it is listening. Eg: 192.168.21.209:6543

fullPathDatabase: the full path of the CA database.

Eg: jdbc:mysql://localhost:3306/markos_analyzer

userName: the user that has access to the database. Eg: root.

password: the password needed to access the database. Eg: root.

maxPoolSize: the maximum number of parallel connection the CA database will accept. Eg: 1000.

Installation and execution

Database installation: Create a mysql db called markos_analyzer and import in it the sql dump markos_analyzer.sql that you find under the db folder in our git repository.

Starting Tomcat: Deploy the CA project in Tomcat. For you convenience you can download the war file for the CA directly from our git repository (in this case just put the file Analyser.war in your webapps folder under the Tomcat installation). Remember to Increase the Java heap size available to Tomcat. To do this, insert in the file startup.bat (Windows) or startup.sh (Linux and Mac) the following command:

JAVA_OPTS='-Xms512m –Xmx4096m -XX:PermSize=512m -XX:MaxPermSize=1024m -Dlog4j.debug=true'

Just Start Tomcat and the CA will be in execution waiting for new projects to analyze from the Crawler.

3. Code Analyser Data Retrieval

Requirements

  • Java 1.7
  • Web application container (e.g., Apache Tomcat 7, GlassFish),
  • Maven 3,
  • An access to Code Analyser Database (MySQL), with db schema aligned with db schema published on 15.04.2014

Download

You can download directly the object code file "markos-code-analyser-data-retrieval-1.3.war" from the "binaries" directory or you can follow the following building process.

Building process

Download and uncompress the source code file "markos-code-analyser-data-retrieval-1.3-sources.zip" from the "sources" directory.

Go to CADR home directory:

cd /markosproject-code-analyser-data-retrieval

Edit configuration file located at:

src/main/resources/config/retrievalConfig.xml

Provide appropriate settings:

  • dbUrl: url of Code Analyser DB MySQL instance, following jdbc url schema (jdbc:mysql://:/). Provide , and
  • user: user name to connect to the aforementioned DB
  • passwd: corresponding password

Run command (in the project home directory, where .pom file is located):

mvn clean package

This builds a WAR archive of the service and places it in the /target directory.

Installation and execution

Deploying at Apache Tomcat:

  • Copy the WAR archive into webapps folder of you web application container (e.g. Apache Tomcat).
  • Run the web application container.
  • By default the service is available at URL: http://localhost:8080/markos-code-analyser-data-retrieval/

Alternative deployment at GlassFish server (using embedded GlassFish for Maven):

  • Run command:

mvn clean install -Pgf

  • By default the service is available at URL: http://localhost:8080/markos-code-analyser-data-retrieval/

4. Repository

Requirements

Java 7 SDK ( http://www.oracle.com/technetwork/java/javase/downloads/)

Download

You can download directly the object code file "markos-repository-1.3.jar" from the "binaries" directory or you can follow the following instructions to build the single sub-subcomponents.

Installation of third party software:

  1. Install Java SDK
  2. Install Apache Tomcat server
  3. Install Owlim SE according to the installation instructions:http://owlim.ontotext.com/display/OWLIMv54/OWLIM-SE+Installation

Building process

Commons

Requirements to build

Building process

Download and uncompress the source code file "markos-commons-1.3-sources.zip" from the "sources" directory.

Go to the markos commons directory, where the repository has been cloned, and use maven to build it.

mvn install

after the previous step the markos commons artifacts will be installed in the local maven repository.

Store

Requirements to build

  • Java 6 SDK - Owlim repository should also work with Java 7 but it wasn't tested with such configuration
  • Apache Maven 3
  • Apache Tomcat 7
  • OpenRDF Sesame 2.7.7
  • Owlim SE 5.4
  • MySQL relational database or other equivalent database accessible via Swing framework

Building Markos Repository

Download and uncompress the source code file "markos-repository-1.3-sources.zip" from the "sources" directory.

Markos Repository component needs: markos commons component

  1. Download Markos Repository source code from sourceforge repository:

http://sourceforge.net/projects/markosproject/files/MARKOS%201.3/sources/

  1. Change directory to reach Markos Repository

cd markos-repository-1.3-sources

  1. Build Markos Repository

mvn install

Now Markos Repository is build and can be used as a library with other markos components. To build software responsible for managing repository lifecycle please follow the instructions:

  1. Change directory to reach Repository Manager

cd repository-manager

  1. Build Repository Manager

mvn package

or

mvn package -Dmaven.test.skip=true to disable tests

The Repository Manager application will be created in the distribution/target directory.

Configuration

Start Apache Tomcat that contains Owlim SE installation

Create repository using following command:

openrdf-sesame/bin/console.sh

Detail information about openrdf-sesame console and repository configuration can be found in http://www.openrdf.org/doc/sesame2/users/ch07.html

Repository must include ontologies used in markos project. The ontologies can be found in markos ontology project. The minimum subset of ontologies can be also found in the repository-manager/controller/ontology directory.

To enable inferencing, a set of inference rules must be defined. The example inference rules used by in the MARKOS project can be found in repository-manager/controller/reasoning/markos_rules_1.3.1.pie file.

The repository_name and repository_url will be required by Browsing and Querying component and for the markos repository controller.

For the markos repository controller provide following information in the controller.properties file: - workspace - directory where data file will be stored - remote.uri - URI of the remote repository that will provide semantic data - remote.id - ID of the repomte repository - remote.uri and repository.remote.id are representing repository created in the previous configuration step.

Other configuration parameters should preserve default values.

Preparing database

Repository manager uses a relational database to store its current state and resources. The Java Swing framework is used to access database content. This means that any relational database supported by the Swing framework can be used as a backend for repository manager. In the MARKOS system Repository manager was tested and used with MySQL database.

The database schema used by the Repository manager can be found in the repository-manager/utils/dbSchema/repository_manager_db_schema.sql file.

The valid connection parameters and location of the file containing SQL queries must be provided in the conf/jdbc.properties file.

In the data_endpoint relation insert a valid url of the Code Analyser Data Retrieval service endpoint.

Execution

Start Apache Tomcat

tomcat/bin/startup.sh

Start markos repository controller process by running

java eu.markosproject.repository.manager.controller.ControllerExampleApp

this application requires single properties file as argument, or use bin/startup.sh script

5. Frontend, Linked data, License analyzer

Requirements

Download

You can download directly the object code file "markos-frontend-1.3.war" from the "binaries" directory and follow the configuration instructions for the single subcomponents or you can follow also the building process instructions for the single sub-subcomponents.

Browsing and Querying

Building process

Download and uncompress the source code file "markos-querying-1.3-sources.zip" from the "sources" directory.

The Markos querying component needs: markos commons component

Any other dependency will be automatically managed by the maven tool.

Use the following maven command to build the component sources

mvn install

after the previous step the markos commons artifacts will be installed in the local maven repository.

Configuration

After the package has been created the component can be configured by creating a property file called querying.properties (it has to be included in a folder called conf included in the runtime environment). The file is a property file customizing the default settings available in the src/main/resources/properties/querying_default.properties file. The relevant settings to customize are the following parameters:

  • url: public url used to connect with the repository server (e.g.http://markos.man.poznan.pl/openrdf-sesame)
  • repository: repository id (e.g. markos_XXX)
  • archive.max_kb: settings to manage the on-disk cache of the downloaded source code archives. If the max size in Kb is less or equal than 0 than the feature is disabled note: the previously cached files are not removed automatically in this case
  • archive.folder: folder in which to store the downloaded archives. Default value = downloads

License checker and analyser

Requirements to build

Configuration

Both the License Checker and the License Analyser requires the same configuration files and ontologies.

Download the marko-ontology project and the Carneades project:

git clone git://git.berlios.de/markos-ontology

git clone https://github.com/carneades/carneades.git

Copy from carneades project config/carneades.clj to ~/.carneades.clj

Note that the file begins with a dot .

Edit ~/.carneades.clj so that :projects-directory points to the projects directory of the Carneades project.

Make a symbolic link from the markos-ontology project to projects/markos/theories/ontologies

Edit the file projects/markos/properties.clj to select the triplestore endpoint and repository name.

License checker

Building process

Download and uncompress the source code file "markos-license-analyser-1.3-sources.zip" from the "sources" directory.

The license checker needs:

  • markos-commons

  • markos-license-analyser

In the markos-license-analyser directory type the following commands:

lein deps

lein jar

If everything compiles correctly you will have a JAR in the target directory.

Installation

If you didn't use Maven or didn't build the the JAR yourself you can download them at:

https://owncloud.fokus.fraunhofer.de/public.php?service=files&t=3f821f1b3029eaa44c9adbd64a82316b

Note that the version provided on the above link may be outdated in comparison to the version provided by the Maven PSNC repository!

Test

Try calling the findCompatibleLicenses method with the following parameters:

"" " http://markosproject.eu/kb/SoftwareRelease/366"

It should returns a list containing:

["http://www.markosproject.eu/ontologies/oss-licenses#GPL-2.0"

"http://www.markosproject.eu/ontologies/oss-licenses#LGPL-3.0"

" http://www.markosproject.eu/ontologies/oss-licenses#LGPL-2.1"]

findSoftwareWithCompatibleLicenses: Not Yet Implemented

License analyser

Installation

Two war files are needed, one for the Carneades Web Service and one for the Carneades Web Application.

Download the two WAR files or see the next section to build the WAR yourself:

https://owncloud.fokus.fraunhofer.de/public.php?service=files&t=3f821f1b3029eaa44c9adbd64a82316b

Building process

git clone https://github.com/carneades/carneades.git

cd carneades/src/CarneadesWebApp

./script/build-war.sh

If everything compiles correctly you will have two WAR files. One in the target directory of the ../CarneadesWebService directory and one in the target directory of the CarneadesWebApp

Deployment with Tomcat

Deploy carneades-web-service-1.0.0-SNAPSHOT-standalone.war with ContextPath /carneadesws

Deploy carneades-webapp-1.0.0-SNAPSHOT-standalone.war with ContextPath /carneades

Test

Try reaching the resources available at these relative paths:

/carneadesws/

/carneades/config

/carneades/#/home

and the URL: http://localhost:8080/carneades/#/license-analysis/introduction/markos?entity=http://markosproject.eu/kb/SoftwareRelease/366

The configuration file is searched in the user.home (Java property) of the process running the Tomcat server. This means that the user.home variable should be defined, if it's not already for the process running Tomcat. Normally this value is automatically set to the user's Unix home but it is not always the case, for instance for Unix service users.

Frontend

Building process requirements

Building process

Install the previous dependences. Markos Frontend component needs:

  • markos querying and browsing component
  • markos license analyser component
  • markos registration component
  • GWT (latest version 2.5.1)

Needed dependencies will be automatically downloaded from current markos maven repository.

You can get the source code in the following page:

https://sourceforge.net/projects/markosproject/files/MARKOS%201.3/sources/

You can also get precompiled war files here:

https://sourceforge.net/projects/markosproject/files/MARKOS%201.3/binaries/

If you choose to compile the source code, unzip the downloaded file.

In the folder src/main/resources/conf there is a configuration file called: querying.properties. This file is used by the application and can be configured to satisfy the conditions reported in the configuration paragraph of the Querying and Browsing component. Note that those very same properties can be edited by the means of the profile described in the pom.xml file.

Now you can use maven to build the web application. This command prepare a WAR file and yes, it takes a while :)

mvn install

Frontend execution

Copy the .war file from the 'target' directory to the webapp directory in Tomcat:

GNU/Linux:

sudo cp target/markos-frontend-1.1.0-20130903-SNAPSHOT.war /var/lib/tomcat7/webapps

In case you use Tomee:

GNU/Linux:

sudo cp target/markos-frontend-1.1.0-20130903-SNAPSHOT.war $TOMEE_DIRECTORY/webapps

After that, you should be able to access to it using the following URL address from your browser:

http://localhost:8080/markos-frontend-1.1.0-20130903-SNAPSHOT/

Linked data execution

Download from the directory "binaries" the file markos-linkeddata-1.3.war and deploy it under Tomcat:

GNU/Linux:

sudo cp markos-linkeddata/target/markos-linkeddata.war /var/lib/tomcat7/webapps

Go to the Catalina directory and create a context file called 'markos-linkeddata.xml' (a template can be found at src/main/resources/webapp/META-INF/context.xml). Change the context parameters:

  • server_base: public url used to publish the Linked Data Access Point (e.g. http://public_host:port/markos-linkeddata)

  • classic_server_base: url of the MarkOS frontend

  • sparql_endpoint: url of the markos repository (e.g. http://markos.man.poznan.pl/openrdf-sesame/repositories/markos_XXX)

  • survey_online: url of the survey used to collect feedbacks from the users.

After that, you should be able to access to it using the following URL address from your browser:

http://localhost:8080/markos-linkeddata/
Source: README.md, updated 2014-07-11