Menu ▾ ▴

Build

Philippe Le Berre consulare abahamne

ESIS FRAMEWORK Developer Guide

General documentation suitable for developpers (java or flex/flash!) needing to install, run, and perform various tasks.

Please read these carefully before asking for help.

Follow this document to have the standard development configuration.

Prerequisites

  • Unix platform (Mac OS X, Ubuntu, CentOS or RedHat (Linux))
  • Java (JDK) (at least 7) from Java (JAVA_HOME must be set)
  • On Mac OS X, the XCode and Developer tools must be installed
  • On other platforms, compilation and build (ie. gcc & co) tools must be present
  • Apache SubVersion (needed for svn ant task) to download the source code
  • A user account (do not use root !), any it will complain

Environment

Mac OS X

Since Mac OS 10.7 (Lion) you'll need a standard OS install and in addition the Developer tools XCode -free- with the command line tools.

Note that subversion is installed by Xcode as part of the command line tools.

To install the Xcode command line tools run the following command :

sudo xcode-select --install


The MacPorts is also need to install autoconf and automake.

By default launchd will impose a limit of 256 opened files, change this to 1024 by adding limit maxfiles 1024 1024 to /etc/launchd.conf.

PostgreSQL requires shared memory, and OS X might not provision enough by default, thus one needs to create or edit the /etc/sysctl.conf and something like

kern.sysv.shmmax=1610612736
kern.sysv.shmmin=1
kern.sysv.shmmni=256
kern.sysv.shmseg=64
kern.sysv.shmall=393216


Linux

CentOS and RedHat

If the server is behind a web proxy, one will need to edit the /etc/yum.conf

proxy=http://mycache.mydomain.com:3128
proxy_username=yum-user
proxy_password=qwerty


To work on ESIS Framework on a CentOS system one need to install the following packages. 

$ yum install gcc
$ yum install gcc-c++
$ yum install expat-devel
$ yum install m4
$ yum install sendmail-cf
$ yum install glib2-devel
$ yum install wget
$ yum install svn


The gcc-java package must be removed to avoid any issues, like an error about and when running ant jni.compile.

$ yum erase gcc-java.x86_64
$ yum erase libgcj.x86_64


Check that the local firewall (iptables) has the port 8080 open, see IPTables for more information.

Ubuntu

Ubuntu has switched to Dash as the default shell, so all our scripts start with #!/bin/bash just to be sure.

One can work on the LTS (no GUI dev).

$ sudo apt-get install openssh-server
$ sudo apt-get install gcc
$ sudo apt-get install g++
$ sudo apt-get install patch
$ sudo apt-get install make
$ sudo apt-get install m4
$ sudo apt-get install ucommon-utils
$ sudo apt-get install libglib2.0-dev
$ sudo apt-get install libc6-dev-x32
$ sudo apt-get install libexpat1-dev
$ sudo apt-get install ttf-mscorefonts-installer
$ sudo apt-get install subversion
$ sudo apt-get install autoconf
$ sudo apt-get install python-dev


Java

The framework requires at least Java 7. USE ONLY THE ORACLE JDK.

Mac OS X

To set the JAVA_HOME environment variable add the following to the user .bashrc:

export JAVA_HOME=$(/usr/libexec/java_home)


Linux (CentOS, RedHat, Ubuntu)

DO NOT USE YUM to install Java, remove any existing JRE or JDK that has been installed with yum.

[root@localhost /] yum list installed | grep java
[root@localhost /] yum erase <whatever jdk or jre that are around>


Download the latest Java JDK (at least 7).

One may also see Java for OS X 2014-001.

[user@localhost ~/]#  wget --no-check-certificate --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u102-b14/jdk-8u102-linux-x64.tar.gz
[user@localhost /]# sudo bash
[root@localhost /]# cd /usr/local
[root@localhost usr]# mkdir java
[root@localhost usr]# cd java
[root@localhost java]# mv ~/jdk-8u102-linux-x64.tar.gz .
[root@localhost java]# gunzip jdk-8u102-linux-x64.tar.gz
[root@localhost java]# tar -xvf jdk-8u102-linux-x64.tar
[root@localhost java]# ln -s jdk1.8.0_102 current


Set the JAVA_HOME, for instance to /usr/local/java/current environment variable by editing the /etc/profile or /etc/bashrc, see 1.

You will need to logout afterwards.

JNI

ESIS Framework has a small JNI (Java Native Interface) part - to fetch the process PID -, therefore the Apache Ant build.xml needs to know where the jni.h header is, and the JAVA_HOME must point to where the JDK is, and not to a generic directory (i.e. /usr versus /usr/java/latest/). If Ant complains about not finding jni.h then check the JAVA_HOME.

Check out the source code

ESIS Framework source code is hosted on a subversion server hosted by Sourceforge.net.

To get through an http proxy one must edit the ~/.subversion/servers file. If you experienced some weird issue SSL, make sure the svn client is using the latest neon library or switch to cerf.

To fetch the current line of code (not recommended unless you have a clue of what you want to achieve, prefer SNAPSHOT tags).

user@host:~$ cd code/esis-framework
user@host:~/code/esis$ svn checkout https://svn.code.sf.net/p/esisframeworks/code/trunk trunk


Setup an APPLICATION (that relies on the ESIS Framework)

Checkout your application code in the app/ directory, for instance:

$ cd trunk/app
$ svn checkout https://svn.code.sf.net/p/esis/code/trunk esis

Edit the my_app_name.conf or have it symlinked to the application one.

Edit the app.properties or have it symlinked to the application one.

[esis-dev ~code/esis-frwk/trunk/app]$ ln -s esis/my_app_esis.conf my_app_name.conf
[esis-dev ~code/esis-frwk/trunk/app]$ ln -s esis/esis.properties app.properties


Create the app main directory.

[esis-dev ~]$ sudo mkdir /opt/<my_app_directory>
[esis-dev ~]$ sudo chown esis-dev /opt/<my_app_directory>


Initialize the working (shell) environment

Before doing anything in the source code directory one must initialize the environment by running . ./env.sh, this will do basic checks ant set all the required environment variables.

YOU WILL HAVE TO DO THIS EVERYTIME YOU START A NEW SHELL !

csmcp@consulare:/opt/esis-framework/trunk$ . ./env.sh
Using the Application csmcp
Using default application configuration conf/app.conf
Using default app db predicate : csmcp
Using default db port localhost
Using default FTP port : 10021
Using default port for Jetty : 8080
Do not forget to install Postgresql with pkgs/
Setting PGDATA to default /opt/csmcp/pgdata
Setting JETTY_HOME
Please install Jetty, see scripts/install_jetty.sh
Setting FLEX_HOME to default
Setting LC_ALL to C
Setting CODE_LINE
Setting MACHINE
Setting ANT_HOME
Updating PATH
Updating MANPATH
Updating LD_LIBRARY_PATH
Setting LIGHTTTPD_PID
Setting FINDBUGS_HOME
Setting GARCHIVEDIR
Exporting EF_APP_XX environment variables
Exporting JETTY_XX environment variables
Exporting EF_APP_CUSTOM_REPORTS_DIR variable
Setting UNAME
ESIS FRAMEWORK: Set-up for /opt/esis-framework/trunk


Open source packages

If behind a proxy server the http_proxy environment variable must be set.

The slow way; BE BRAVE download source code and build the files. The build process is based upon GSRC, whicu used to be GAR, it might prove useful to read a little bit about it before moving forward, as O/S vendors got us used to change pieces there and there that breaks GSRC build process.

To optimize the binaries for your current plaform create a file flags-{results of running the command uname -n} and put it in the esis/main/pkgs/autoinstall directory. In this file sets the CFLAGS and CXFLAGS to optimize the binaries. Doing this can lead to dramatic performance improvement for Postgresql. Some sample files are provided in the esis/main/pkgs/autoinstall directory.

Cleanup the PATH environment variable, especially on OS X if MacPort, Fink of Brew is installed.

The recommended way for the newcomers :

$ . ./env.sh
$ cd pkgs
$ ./build.sh
$ make -f PkgMakefile clean
$ make -f PkgMakefile paranoid-install


Perl

All the Perl scripts points to /usr/bin/perl. So before going any further do a perl -v to ensure that the one in /usr/bin/perl is the one you want to use, if not them symlink the right one to /usr/bin/perl. You can rename the 'old' to perl5.

Install CPAN modules

You should be ready to compile now. In order to test fully you'll need to install some perl modules, the easiest is to use CPAN with the command sudo perl -MCPAN -e shell.

$ sudo perl -MCPAN -e shell
cpan> upgrade
... upgrading all existing perl package ...
cpan> reload cpan
cpan> install Bundle::CPAN
cpan> install LWP
cpan> install DBI
cpan> install DBD::Pg
Path to pg_config: /opt/ESIS/bin/pg_config
cpan> install MIME::Base64
cpan> install MIME::Tools
cpan> install Time::HiRes
cpan> install Digest::MD5
cpan> install Unicode::String
cpan> install Error
cpan> install Devel::Symdump
cpan> install Class::Inner
cpan> install XML::Parser
cpan> install Class::Inspector
cpan> install SOAP::Lite
cpan> exit


In case of issues, notably with proxy, do not hesitate to delete the .cpan directory, and once in the CPAN shell to execute o conf init. If it fails to grab a MIRRORED.BY file than in directory .cpan/sources use wget http://www.cpan.org/MIRRORED.BY, somehow it gets through proxies better.

Postgres database configuration

After this, you have installed postgresql. To run postgresql as your user you'll need to create a data directory for it, setup your environment, create a database, and create a user with access to this database.
First ensure you have set the shmen as explained in the Wiki on System Configuration.

Second, if you intend to use a specific location then you need to set the PGDATA environment variable, otherwise it will point to the default /opt/ESIS/pgdata.

Then proceed with initializing the database :

user@host:~$ cd code/esis-framework/trunk
user@host:~/code/esis-framework/trunk$ . ./env.sh
user@host:~/code/esis-framework/trunk$ scripts/pg_init.sh
user@host:~/code/esis-framework/trunk$ scripts/reinit_db.sh
Postgres configuration changes


The error could not create shared memory segment means that one has to reduce the shared buffer size in the /opt/MY_APP/pgdata/postgresql.conf, or allow more shared memory (which might not be possible without adding more RAM).

The default postgresql.conf is kept as postgresql.conf.orig, see sql/config for the ones that are used.

For more information on how to tune postgresql.conf check :

Setup Apache Ant, Jetty and Apache FtpServer

For the build we need also to setup some environment variables, Apache Ant, Jetty and Apache FtpServer.

user@host:~/depots/ESIS/main$ . env.sh 
user@host:~/depots/ESIS/main$ make init
ESIS set-up started

Creating the local temp directory .temp/build
ESIS - ant install
Removing previous version
Extracting ant...
Done.

Installing Jetty (8.1.7.v20120910)

Removing previous installation

Extracting Jetty...

Deleting unnecessary web applications

Now configuring Jetty

Now configuring ESIS application

Done.

ESIS ftp install

Removing previous installation

Extracting ftpserver...
tar: A lone zero block at 5578

Now configuring local data for ESIS unit tests

Done.
Buildfile: /code/esis/main/build.xml
  [taskdef] Could not load definitions from resource net/sf/antcontrib/antcontrib.properties. It could not be found.
  [taskdef] Could not load definitions from resource cpptasks.tasks. It could not be found.
  [typedef] Could not load definitions from resource cpptasks.types. It could not be found.
  [taskdef] Could not load definitions from resource flexTasks.tasks. It could not be found.

init.pre-init:
     [copy] Copying 1 file to /code/esis/main/.build/apache-ant-1.8.4/lib
    ...
BUILD SUCCESSFUL
Total time: 0 seconds
ESIS set-up finished


The Ftp Server is used for testing the ftp mirroring in probes, it is listening on port 10021.

Jetty is used by EsisDaemon, use the scripts/daemon [start|stop] command, by default it listens on port 8080.

Prepare build environment

There's an ant target that does all this, look in the build.xml for more details about what it does.

user@host:~/depots/ESIS/main$ ant setup


Build ESIS and run tests

There's an ant target that does all this, look in the build.xml and in the build/ directory for more details about what it does.

user@host:~/depots/ESIS/main$ ant debug-dist
user@host:~/depots/ESIS/main$ ant test.db
user@host:~/depots/ESIS/main$ ant test.ef
user@host:~/depots/ESIS/main$ ant doc.junit


To learn about all the Ant tasks:

user@host:~/depots/ESIS/main$ ant -p

If this finishes successfully then all is well with your system! Congratulations!
You can browse your local unit test results by looking at the bookmarks.html file in the doc directory.

Only if you have all the required test environment you can give a shot at

user@host:~/depots/ESIS/main$ ant smoke


There is also a suite of Perl Unit tests to test the esis command line. The tests are in scripts/Test. To run one scripts/TestRunner.pl [test name like Test::cli::locations]

To run a single test :

user@host:~/depots/ESIS/main$ ant test.t -Dt=net.sourceforge.esisframework.test.test05util.test08PasswordChecker


To get the list of Ant tasks

ant -p

Debugging and Log files

The ESIS Framework log files are in the directory /opt/MY_APP/var/log/ to change the log settings and turn on advanced debugging options one can edit the app.properties file.

ESIS use the Apache log4j library and the lowest log level is TRACE.

Setting up Fonts for JasperReports

On OS X check the file /opt/ESIS/share/ESIS/reports/styles.jtrx which contains the JasperReports style sheet that standard reports use, and turn the Arial_Normal_OSX into Arial_Normal otherwise you might have an error like :

ERROR : Could not load the following font : 
pdfFontName   : arial.ttf
pdfEncoding   : UTF-8
isPdfEmbedded : true


See:

Some helpful tools

  • MindTerm a Java SSH client with a "Free Edition"
  • Jasper iReport designer to have a GUI to edit JasperReports.
  • Squirrel SQL is something you'd like to have handy.
  • A custom sqsh to have a command line for MS-SQL (yeah !) might prove useful.
  • Version is cool GUI for subversion on OS X.
  • HTTPScoop is cool to monitor HTTP traffic on OS X.
  • BBEdit a cool code editor on OS X.
  • PGCluu a performance monitoring for Postgresql.
  • PGtoolkit Tools for PostgreSQL maintenance.
  • PGCookbook set of articles, scripts and configuration files made to help with PostgreSQL maintenance.
  • PGAdmin a popular and feature rich Open Source administration for Postgresql

Related

Wiki: Custom-Report
Wiki: Fonts-Report
Wiki: Home

Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.