NTER Executable Code Repository Setup
Setting Up an NTER Code Repository
Table of Contents
1. Introduction
1.1. Overview
1.2. Maven Projects
1.3. Process of submitting Code
1.4. Continuous Integration
1.5. Code NOT on the Repository
2. Google Code
2.1. Overview
2.2. Java
2.3. Python/PubSubHubbub
2.4. PHP
2.5. Shell Scripts
2.6. Configuration Scripts
3. Setting Up A Code Repository
3.1. Overview
3.2. Hudson and Jenkins
3.3. Repository.nterlearning.org
3.4. Amazon Simple Storage Service
4. Deployment of Production Code
4.1. Hub Install
4.2. Manual Install
5. Glossary
1. Introduction
1.1 Overview
The NTER Federated Learning Management System consists of numerous open source software projects and custom code written to implement the system. There are servers that provide Portal Services, Course Catalog Registry, Searches, a Learning Management System (LMS), Single Sign On (SSO), Atom Feeds of student progress, etc.
The custom code is stored in a revision control system on Google Code. Modifications to the custom code are submitted to the NTER Project on Google Code. When the administrator of the project merges changes into the main branch of the code, a service on the code Repository (repository.nterlearning.org in our case) detects the presence of new code and compiles it.
The compiled code is then either sent to the production servers automatically, or manually installed by system administrators.
This guide is primarily intended to describe the Code Repository system. The Code Repository (Repo) stores compiled code in the case of Java and production ready code scripts in the case of PHP code, Python Code, and Shell Scripts.
The Repo stores multiple versions of the code so that any version of a particular program can be installed.
In the case of the PHP code for the NTER LMS the code is compiled into a Java based installer called IzPack. This creates an executable Java jar file that can be run on the desktop or the command line. IzPack and Java are cross platform, so the same jar file will install the LMS onto a windows or Linux server.
1.2 Maven Projects
Most of the code modules in the NTER System are written as a Maven Projects. Maven is a more sophisticated tool than make or Ant, and has the big advantage of defining the structure of the project so that it can be downloaded in to multiple development environments.
Before Maven each development environment had its own file and directory format which made it difficult to move a project from one IDE to another. Most IDEs now support Maven or have Maven plugins.
Maven project modules in the NTER System can be downloaded into most development systems and modified without regard to the IDE originally used to create the code.
1.3 Process of submitting Code
The Google Code source code repo has an administrator for the NTER project who is responsible for moving source code enhancements into the main branch for use in production systems. If you would like to work on code, contact the administrator on the Google Code site to be made a contributor.
After you submit code the administrator will consider it for use in the main branch.
1.4 Continuous Integration
After source code is moved into the main branch on the Google Code NTER Project, the Continuous Integration system on repository.nterlearning.org retrieves the source code and compiles it, if necessary. The ready to run code is then available on the code repository for deployment to production servers.
See section 4 of this document for more details on deployment.
1.5 Code NOT on the Repository
There are several open source packages used in the NTER System that are commonly available that are not on the Code Repo or the Source Code Repo. Among these are the MySQL Database, the Apache Web Server, the PHP Interpreter that runs under Apache, the Apache Tomcat Servlet Container, the Shibboleth Single Sign On (SSO) Identity Provider, Secure Socket Layer (SSL) software, the Java Development Kit (JDK), the Hudson and Jenkins continuous integration servers, among others.
These programs typically must be installed for a particular operating system.
One exception to this is the LifeRay Portal Server. NTER installs a particular version of LifeRay to be sure that the custom Portlets written for LifeRay will function properly.
LifeRay, in turn, installs its own copy of the Apache Tomcat Servlet Container to insure that LifeRay functions properly with it.
2. Source Code Storage
2.1 Overview
Source code is stored on the NTER Project Site on Google Code (https://code.google.com/p/nter/source/browse/). This is the human readable form of the code before compiling in the case of Java, and the text files containing code for the interpreted languages PHP, Python, and Shell Scripts.
The source code is “pushed” to the Google Code NTER project using the Mercurial Revision control System.
If you wish to modify code you can download the code using Mercurial (Hg) from your command line or with Mercurial plugins for your development system. Most integrated development environments (IDE’s ) support Mercurial. Eclipse and NetBeans both do so. Mercurial is an open source product. For details see: http://mercurial.selenic.com/
2.2 Java
Java source code is used to write LifeRay Portlets that implement the Course Catalog, Search Engine, and Portlets that automatically update the NTER Portlets to new versions.
The Portlets may contain code to set up or modify the MySQL database used by LifeRay to add new features, or modify old ones.
The Shibboleth Identity Provider server also runs Java web apps that provide SSO services.
The Search Server uses the Java based Lucene search engine with the Java Servlet based SOLR web front end.
Although these programs are not store on the Code Repo, many configuration files used to install them are on the Code Repo.
2.3 Python/PubSubHubbub
The code that is used to track student progress is written in Python and runs on a PubSubHubbub server. This is “server push” software that sends out student progress updates to the Portal servers. This is more efficient for the rapidly changing student progress data than requiring the Portals to constantly poll the LMS servers for student progress data.
2.4 PHP
The NTER LMS is a written in PHP. PHP is an interpreted programming language that does not get compiled into source code.
To make it easier to manage the installation of the hundreds of PHP code files, it is packed into an IzPack self extracting archive. IzPack uses Java to perform this task. Consequently, the NTER LMS installer is a Java jar file
2.5 Shell Scripts
Numerous shell scripts are used to perform tasks in the NTER System. Mostly they are used by the Hub installer to install Portals and LMS serves for institutions setting up NTER.
Most of the Shell Scripts are compiled into Java code. They can be found on the Google Code source code repo.
Stand alone Shell Scripts are also available that are used to perform an installation of the LMS and Portal servers from the command line.
2.6 Configuration Files
Numerous configuration files, or portions of files, are embedded the shell scripts. These are used to set up Apache, Tomcat, MySQL, PHP, and Shibboleth on the Portal and LMS servers.
3. Setting Up A Code Repository
3.1 Overview
Setting up a code repository requires that your code modules be available on the network to servers that might want to install them. This task is made easier by tools like a continuous integration server. For the nterlearning.org system Hudson and Jenkins are used. Section 3.2 below describes them more fully.
There are other servers available and you necessarily have to use Hudson or Jenkins. Even if you do, you will have to carefully modify the setting compared to ours to point at your source code repository.
You will also need to modify the NTER install scripts to point at your executable code on your repo, instead of ours.
3.2 Hudson and Jenkins
There is often confusion about Hudson and Jenkins. The Hudson project became proprietary and Jenkins was forked as an open source project. Hudson eventually went to open source as well but Jenkins is by far the more popular now.
Jenkins can be obtained at: http://jenkins-ci.org/. There is extensive documentation on the site, as well, covering the installation, setup, and modification of the software.
Jenkins is a Java based web app and can be run under Tomcat or other Servlet containers.
3.3 Repository.nterlearning.org
Repository.nterlearning.org is the code repository for the nterlearning.org system. The code stored there can be accessed by URL’s and installed on remote servers.
The code modules are called artifacts and the site is referred to as an Artifactory.
When you browse to the repository.nterlearning.org site and examine the code artifacts, you will see a URL in the “General” tab.
If you click on the URL you can download any of the files to your computer. In a Linux environment you can use utilities like “wget” along with that URL to download the file to your Linux computer.
You are free to use Artifactory code to install your NTER servers. If you want to make your own code repo, you will need to set up your own servers and modify the install scripts to point at them.
Some code for NTER is not stored on the code repository at repository.nterlearning.org. It is stored on Amazon S3. The next section will explain more.
3.4 Amazon Simple Storage Service
Amazon Simple Storage Service, popularly known as Amazon S3, is a low cost file storage utility provided by Amazon for users of it Amazon Web Services (AWS) cloud service.
Files stored on S3 can only be accessed by a URL, but it is handy storage for files that don’t change often.
Since we install most NTER servers on AWS and S3 is inside Amazon’s firewall, it is a quick way to load large files on servers.
In the case of nterlearning.org files like the LifeRay program are stored there. Those files almost never change. The Portlets for LifeRay that change frequently are stored on Artifactory.
4. Deployment of Production Code
4.1 Hub Install
The hub installer is a web application for easy installation of Portal and LMS servers on AWS Cloud virtual machines.
It is documented elsewhere, but in its internal scripts it pulls code for the installation from Artifactory and from Amazon S3.
4.2 Manual Install
NTER can also be installed with Linux shell scripts manually. The scripts are currently only for Ubuntu Linux, but should be easy to modify for other versions.
The scripts install the Portal and LMS servers on your local systems and link your new installation to the central NTER login, registry, search, and PubSubHubbub servers.
General install instructions can be found on the NTER SourceForge site at:
https://sourceforge.net/p/nter/wiki/search/?q=install&project=0
The script to install the Portal can be found on the NTER SourceForge site at:
https://sourceforge.net/p/nter/wiki/NTER%20Portal%20Manual%20Install/
5. Glossary
Ant-
A build tool written in Java that is similar to “make” for C language programs.
Atom Feed-
A feed similar to RSS that will stream frequently updated data to an Atom consumer.
AWS-
Amazon Web Services. Amazons cloud service of virtual machines that are used for the nterlearning.org NTER setup.
Continuous Integration-
An automated system for compiling and deploying changes to program code and deploying to production environments. Hudson and Jenkins are example tools.
Hub Installer-
A web based tool used to install NTER Portals and LMS servers onto Amazon Web Services Virtual Machines.
Hudson-
A continuous Integration tool.
IDE-
Integrated Development Environment. A tool used to write, compile, test, and deploy software.
JAR file-
Java ARchive. This is Java code and documents that are compressed into a zip file with a manifest to tell java how to unpack and use the files.
Jenkins-
A continuous integration too that is a spinoff of Hudson.
LMS-
Learning Management System. In the case of the NTER System, the LMS component is called NTER
LMS. It is written in PHP and is a modification of the Ilias LMS. The modifications allow it to share courses with students at other sites.
Maven-
A software build automation tool that incorporates the project structure so that it can be opened in any Integrated Development Environment (IDE) like Eclipse, NetBeans, etc., that supports Maven. Used in place of tools like Ant and make.
Mercurial-
A cross platform revision control system. It uses hg, the atomic symbol for Mercury, as its shell command.
Portlet-
A small java program that runs in a Portal service. NTER uses the LifeRay open source
Portal. The custom Portlets written for NTER are loaded into the Portal for use on the student portal.
PubsubHubbub-
PubSubHubbub is a server that pushes data out to clients. It is used for data that frequently changes.
It is more efficient for rapidly changing data than a traditional polling server like RSS or an Atom feed.
Revision Control System-
A system used to track changes in files. Frequently used with software projects to track changes in source code. Mercurial and Git, are examples.
RSS Feed-
Rich Site Summary. A protocol for a feed of frequently updated data.
Servlet-
A small Java program used to add a capability to a server.
Servlet Container-
A program that runs Java Servlets. Examples are Tomcat and Jetty, among others.
Single Sign On-
The ability to login to multiple different servers in a system while only having to enter your
user name and password once.
SSL-
Secure Socket Layer. Software that provides a secure web connection from the server to the browser.
Typically Open SSL is used with NTER.
SSO-
Single Sign On. See Above.
