Recent changes to TechnicalManual14

TechnicalManual14 modified by Aleksi Kallio

Aleksi Kallio — Fri, 11 Jul 2014 12:31:48 -0000

--- v21
+++ v22
@@ -1,6 +1,7 @@
-[TOC]
-
 # Technical manual for Chipster 1.4
+
+> **Note! This is an unmaintained archive site.**
+> Wiki has been moved to Github and current documentation is available at [https://github.com/chipster/chipster/wiki](https://github.com/chipster/chipster/wiki)

 The manual covers Chipster platform version 1.4 and older. It instructs in setting up your own Chipster server, adding your own tools into Chipster, and more. For the user manual, please see https://extras.csc.fi/biosciences/chipster-manual/.

TechnicalManual14 modified by Aleksi Kallio

Aleksi Kallio — Thu, 26 Jun 2014 06:42:43 -0000

--- v20
+++ v21
@@ -38,11 +38,11 @@

 After downloading extract the tar archive. It contains directory "chipster", where all components are in their own subdirectories. It can be placed anywhere, but usually **/opt/chipster** is used. 

-Downloading and extraction can be done easily on command line (we use version 1.1.2 here): 
+Downloading and extraction can be done easily on command line (we use version 1.4.7 here): 

     cd /opt
-    wget http://chipster.csc.fi/downloads/chipster-1.1.2.tar
-    tar xf chipster-1.1.2.tar
+    wget http://www.nic.funet.fi/pub/sci/molbio/chipster/dist/versions/1.4.7/chipster-1.4.7.tar.gz
+    tar -xzf chipster-1.1.2.tar.gz

 **2) Installing external tools**

TechnicalManual14 modified by Aleksi Kallio

Aleksi Kallio — Thu, 26 Jun 2014 06:42:42 -0000

--- v19
+++ v20
@@ -185,8 +185,8 @@

   * Download Weeder (v. 1.3) from http://www.pesolelab.it/index.php?mod=04_Tools 
   * Install to directory weeder/Weeder1.3 in Chipster tool directory 
-  * Download and unpack sequence files to directory weeder/seqs: http://www.nic.funet.fi/pub/sci/molbio/chipster/dist/tools-extras/weeder-seqs/all-sequence-files.zip 
-    * For information on how the sequence files were generated, see http://www.nic.funet.fi/pub/sci/molbio/chipster/dist/tools-extras/weeder-seqs/README.TXT 
+  * Download and unpack sequence files to directory weeder/seqs: http://www.nic.funet.fi/pub/sci/molbio/chipster/dist/tools_extras/weeder_seqs/all-sequence-files.zip 
+    * For information on how the sequence files were generated, see http://www.nic.funet.fi/pub/sci/molbio/chipster/dist/tools_extras/weeder_seqs/README.TXT 

 #### Muscle (old)

TechnicalManual14 modified by Aleksi Kallio

Aleksi Kallio — Thu, 26 Jun 2014 06:42:42 -0000

--- v18
+++ v19
@@ -718,11 +718,11 @@

 ### Making compute service aware of the script

-It is possible to override scripts running at a compute service for making fixes, local modifications etc. You need to create a directory _comp/custom-scripts/R_ and place a script with the correct file name to that directory (from example, stat-two-groups.R for Two groups test). Next time the script is used the compute service will read the updated script and use it instead. 
+It is possible to override scripts running at a compute service for making fixes, local modifications etc. You need to create a directory **comp/custom-scripts/<RUNTIME TOOL PATH>**, where **<RUNTIME TOOL PATH>** matches the **toolPath** parameter of the corresponding runtime in **comp/conf/runtimes.xml** (before version 1.3.0, the correct path is always **comp/custom-scripts/R**). Then place a script with the correct file name to that directory (from example, stat-two-groups.R for Two groups test). Next time the script is used the compute service will read the updated script and use it instead. 

 To check the script file names of different analysis tools, you have to unzip chipster-x-x-x.jar. In future this functionality will be made a lot more elegant with a new analysis module system. 

-Current limitations (version 1.2.3): 
+Current limitations: 

   * overriding scripts changes the order of analysis tools in client (overridden scripts will be last) 
   * when adding new scripts (not overriding existing), compute service must be rebooted for them to become visible

TechnicalManual14 modified by Aleksi Kallio

Aleksi Kallio — Thu, 26 Jun 2014 06:42:41 -0000

--- v17
+++ v18
@@ -574,7 +574,7 @@
   * write a description, so that the script can be run and shown in the client application 
   * make compute service aware of the script

-You should also follow [conventions for Chipster analysis tools](ChipsterToolConventions). 
+You should also follow [conventions for Chipster analysis tools](#Tools_conventions). 

 ### Making R scripts Chipster compatible

@@ -766,19 +766,10 @@

 The goal in Chipster is always to produce a coherent user experience. Here are some conventions that can be useful when integrating tools into Chipster and should be followed when writing tools that are to be integrated into Chipster main repository. 

-### Microarray analysis module conventions
-
   * The default data format is TSV (tab separated values), with one row for each gene or probeset 
   * The first column should be unnamed or "identifier" and contain the gene/probeset name 
   * Tool should not removed any existing columns unless the row structure is changed 
     * In other words, inputs can have annotation etc. data that just passes through analysis steps 
-  * See AnalysisToolInputsAndOutputs for more information, in version 2.0 typing will be decided by the analysis module and will be described in this page 
-
-### NGS analysis module
-
-  * Tools should accept and produce read data in BAM format 
-  * Tools should assume chromosome names to not have any prefices or suffices (so "12" instead of "chr12") 
-    * Though some prefixes and suffices are supported, but for clarity their use is not encouraged 

 # FAQ

TechnicalManual14 modified by Aleksi Kallio

Aleksi Kallio — Thu, 26 Jun 2014 06:42:40 -0000

--- v16
+++ v17
@@ -590,7 +590,7 @@

 ### Writing VVSADL header

-R-scripts must be described by using a Chipster specific description format called VVSADL (Very Very Simple Analysis Description Language). A VVSADL-snippet is added in comments as a header to a R-script file. The specification for the VVSADL can be found from the Javadoc documentation of the class [[http://chipster.csc.fi/javadocs/1.4/f.../VVSADLSyntax.html|fi.csc.microarray.VVSADLSyntax](http://chipster.csc.fi/javadocs/1.4/fi/csc/microarray/description/VVSADLSyntax.html|fi.csc.microarray.VVSADLSyntax)]. 
+R-scripts must be described by using a Chipster specific description format called VVSADL (Very Very Simple Analysis Description Language). A VVSADL-snippet is added in comments as a header to a R-script file. The specification for the VVSADL can be found from the Javadoc documentation of the class [[http://chipster.csc.fi/javadocs/1.4.0.../VVSADLSyntax.html|fi.csc.microarray.VVSADLSyntax](http://chipster.csc.fi/javadocs/1.4.0/fi/csc/microarray/description/VVSADLSyntax.html|fi.csc.microarray.VVSADLSyntax)]. 

 Here is an example of a VVSADL snippet to attach before of the actual R code: 
@@ -764,7 +764,7 @@

 ## Tools conventions

-Here we describe different conventions that should be followed when developing or integrating tools to Chipster. The goal in Chipster is always to produce a coherent user experience, for which reason a lot of attention should be placed to harmonising the tools. 
+The goal in Chipster is always to produce a coherent user experience. Here are some conventions that can be useful when integrating tools into Chipster and should be followed when writing tools that are to be integrated into Chipster main repository. 

 ### Microarray analysis module conventions

@@ -780,43 +780,6 @@
   * Tools should assume chromosome names to not have any prefices or suffices (so "12" instead of "chr12") 
     * Though some prefixes and suffices are supported, but for clarity their use is not encouraged 

-# System development
-
-## API documentation
-
-API documentation for Chipster code base is available here: 
-
-http://chipster.csc.fi/javadocs/ 
-
-  
-
-
-## Getting the source code
-
-Chipster source code is stored in a Mercurial repository, hosted at the Google Code http://code.google.com/p/chipster/. 
-
-Anyone can get a local copy of the repository by installing a Mercurial client and cloning the repository: 
-    
-    
-    hg clone https://chipster.googlecode.com/hg/ chipster
-    
-
-The latest working version of the system is in the default branch. You can update your repository with latest changes from the main repository by pulling changes from there. 
-
-For instruction on getting started with Mercurial: http://mercurial.selenic.com 
-
-## Committing changes
-
-The beauty of Mercurial version control is that you can have you own repository for working the code independently of the main repository. After you have developed something that you would like to be merged to Chipster main repository, do the following: 
-
-  * pull changes from main repository: https://chipster.googlecode.com/hg/ 
-  * merge your code to tip of the default branch 
-  * write unit tests and run them 
-  * carefully test everything through the client also 
-  * send email to chipster@csc.fi and we will pull changes from your repository into the main repository 
-
-If you would be interested in working with some new cool features with Chipster, you are always welcome to first drop as an email. We will be very glad to help! 
-
 # FAQ

 ## Latest version of Chipster
@@ -842,7 +805,7 @@

 **Q:** Starting Chipster server environment results in: "Could not detect hardware architecture, please set platform manually."  
-**A:** If hardware architecture is not detected automatically, it can be set manually by editing **chipster/shared/bin/chipster-generic.sh**. Architecture is configured by changing the PLATFORM line to match you hardware architecture (see comment above the line for options). 
+**A:** If hardware architecture is not detected automatically, it can be set manually by editing all instances of **chipster-generic.sh**. Architecture is configured by changing the PLATFORM line to match your hardware architecture (see comment above the line for options). 

 **Q:** I get "RSA premaster secret error" when trying to run Chipster server.

TechnicalManual14 modified by Aleksi Kallio

Aleksi Kallio — Thu, 26 Jun 2014 06:42:40 -0000

--- v15
+++ v16
@@ -54,16 +54,16 @@

 For more information on setup tool see [Setup tool section](#Tool_installation_in_Linux). 

-Some tools require additional dependencies and fulfilling them is the most difficult part of the installation. However those steps can be skipped and done later, as only few of the analysis tool require additional dependencies to be met. The setup tool will print out instruction for carrying out the remaining installation steps for additional tools and databases. For more information on them see [InstallingExternalApplications]. 
+The setup tool will print out instruction for carrying out the remaining installation steps for additional tools and databases. 

 **3) Configuring Chipster services**

-To configure the Chipster services, run the following to scripts. Both scripts will ask for confirmation before writing changes to files. Defaults should be fine for a local installation. 
+To configure the Chipster services, run the following two scripts. Both scripts will ask for confirmation before writing changes to files. Defaults should be fine for a local installation. 

     ./configure.sh
     ./genpasswd.sh

-configure.sh configures all the components, and genpasswd.sh generates secure passwords that server components use to authenticate each other. Please note that if you distribute Chipster to several machines you have to manually set passwords. 
+**** configure.sh** configures all the components, and **genpasswd.sh** generates secure passwords that server components use to authenticate each other.**

 **4) Starting and stopping service**s 

@@ -71,22 +71,22 @@

     ./chipster start

-In addition to 'start', 'stop', 'restart', and 'status' can also be used. 
+In addition to **start**, you can also use **stop**, **restart**, and **status**. 

 **5) Testing installation**

+To start the client using Java Web Start, go to the Web Start address specified when running the **configure.sh**. Default address is: 
+
+http://hostname-of-this-machine:8081 
+
+**Note!** Java Web Start server (Jetty) is not bundled to backported versions 1.1.x. You have to set up your own web server for serving Web Start files. 
+
 To start the client locally (on the same machine as the services), run: 

     ./client/bin/chipster-client

-To start the client from other machines using Java Web Start, go to the Web Start address specified when running the configure.sh. Default address is: 
-
-http://hostname-of-this-machine:8081 
-
-**Note!** Java Web Start server (Jetty) is not bundled to backported versions 1.1.x. You have to set up your own web server for serving Web Start files. 
-
-The default username/password is chipster/chipster. Users can be added by editing the userlist at auth/nami-work-files/users. Chipster also supports a host of other more advanced authentication providers. 
+The default username/password is chipster/chipster. Users can be added by editing the userlist at **auth/security/users**. Chipster also supports several more advanced authentication providers. 

 **6) Starting services at boot time**
@@ -112,16 +112,16 @@

   * We have ready made environments available for some platforms 
   * We have specified exactly what is needed to run Chipster (see **chipster/comp/conf/environment.xml**) 
-  * We do not require any of the external dependencies to be there, you just wan't be able to use some of the tools 
+  * We do not require any of the external dependencies to be there, you just won't be able to use some of the tools 
   * We have written [a little tool](SetupTool) that does most of the installation automatically (**setup.sh**) 
-  * We try to keep dependencies to external applications to minimum 
+  * We try to keep dependencies to minimum 
   * And we have tried to document everything 

 And of course we are constantly improving the support for external applications. Feedback and suggestions are always welcome. 

 ### What do "tools" mean?

-By external applications we mean the computational environment needed to run Chipster compute service. Chipster itself is plain Java and does not have any dependencies to external application other than Java Runtime Environment. We do package Chipster with Tanuki Software's free Java Service Wrapper for convenience, but using the wrapper is not required. So, without the external applications in place your compute service will boot up, but will not be able to run successfully any actual analysis jobs. 
+By external applications we mean the computational environment needed to run Chipster compute service. Chipster itself is plain Java and does not have any dependencies to external application other than Java Runtime Environment. We do package Chipster with Tanuki Software's free Java Service Wrapper for convenience, but using the wrapper is not required. So, without the external applications in place your compute service will boot up, but will not be able to run successfully any analysis jobs. 

 External dependencies can be divided to 3 layers. 

@@ -129,11 +129,11 @@
   2. external applications and databases (R and others) 
   3. R packages 

-Level 1 contains a collection of operating system packages that are required for applications at levels 2 and 3 to work. Naturally level 1 is OS specific and so the packages are installed into OS specific locations using OS specific tools (typically **apt-get** or **yum**). Levels 2 and 3 are contained in the **Chipster tools directory**. The most important application at level 2 is **R**, as it hosts most of the analysis functionality and is also the basis for layer 3. Rest of the external applications are explained in more detail in [#More_information_on_external_applications]. There are also some simple databases, i.e. plain files, that reside on layer 2. The R specific layer 3 consist mostly of CRAN and Bioconductor packages, with some additional third party packages. They are installed using the standard R installation methods and will be located in **chipster/tools/R-<version>/library**. Setup tool should handle layer 3 automatically without trouble. 
+Level 1 contains a collection of operating system packages that are required for applications at levels 2 and 3 to work. Naturally level 1 is OS specific and so the packages are installed into OS specific locations using OS specific tools (typically **apt-get** or **yum**). Levels 2 and 3 are contained in the **Chipster tools directory**. The most important application at level 2 is **R**, as it hosts most of the analysis functionality and is also the basis for layer 3. Rest of the external applications are explained in more detail in [next section](#More_information_on_external_applications). There are also some simple databases, i.e. plain files, that reside on layer 2. The R specific layer 3 consist mostly of CRAN and Bioconductor packages, with some additional third party packages. They are installed using the standard R installation methods and will be located in **chipster/tools/R-<version>/library**. Setup tool should handle layer 3 automatically without trouble. 

 Chipster tool directory or tool home is the place to store all external dependencies (except for OS packages). By default it is **/opt/chipster/tools** (since version 1.4.0, earlier versions had a non-standardised approach). Analysis scripts have access to tool directory path via a variable so that they can access external applications and databases. You need to configure tool home to **chipster/comp/conf/runtimes.xml** if you change it. 

-### Basic installation
+### Basic installation with setup.sh

 It is advisable to first install R. After that check that the R installation directory corresponds to that in **chipster/comp/conf/environment.xml** and edit the file if needed. 

@@ -234,63 +234,64 @@

 Chipster server has experimental support for Windows. As the bioinformatics tool environment is Unix oriented, doing a complete installation in Windows will require significant efforts. 

+  
+
+
 # System administration

 ## Chipster architecture

-The shortest description for Chipster architecture would be that it is very flexible. The Chipster environment is based on Message Oriented Architecture. Components are connected using message broker (ActiveMQ). This results in a loosely coupled distributed system. Chipster is designed to be based on the idea of broadcast, allowing components to be unaware of each other. Also the system does not depend on the protocol used for communication. 
+The shortest description for Chipster architecture would be that it is very flexible. The Chipster environment is based on message oriented architecture (called also message passing architecture or message oriented middleware architecture). Components are connected using message broker (ActiveMQ). This results in a loosely coupled distributed system. Chipster is designed to be based on the idea of broadcast, allowing components to be unaware of each other. Also the system does not depend on the protocol used for communication. 

 The Chipster environment consists of the following components: 

   * message broker (1 to many) 
-  * file server (1) 
+  * file broker (1 to many) 
   * authenticator (1) 
   * compute server (1 to many) 
   * client (many) 

-All components can be added and removed on fly. On case there are multiple instances of a same component running there is no need for extra configuration, because for example multiple analysers can function without being aware of each other. This allows system administrator to add analyser components on fly if there is need for extra processing power, for example during large courses. 
-
-Currently there can be only one file server and authenticator. However both of these limitations are to be removed. Duplicating file server requires very small modifications, authenticator a bit more. 
-
-One of the key ideas in designing Chipster architecture was to carefully consider where each bit of the system's state is managed. Chipster client follows fat client paradigm where client is functionally reach. This decision was made to keep server environment simple and lightweight, reduce number of messages, distribute processing load (especially data visualisation) to clients and allow improved user experience as client application is mostly independent of server components. As most of the relevant state has the same lifecycle as one client session, managing state at the client side is also logically a good solution. 
+All components can be added and removed on fly. In case there are multiple instances of a same component running there's no need for extra configuration, because, for example, multiple analysers can function without being aware of each other. This allows system administrator to add analyser components on fly if there is need for extra processing power, for example during large courses. Currently there can be only one and authenticator. 
+
+One of the key ideas in designing Chipster architecture was to carefully consider where each bit of the system's state is managed. Chipster client follows fat client paradigm where client is functionally rich. This decision was made to keep server environment simple and lightweight, to reduce number of messages, to distribute processing load (especially data visualisation) to clients and to allow improved user experience as client application is mostly independent of server components. As most of the relevant state has the same lifecycle as one client session, managing state at the client side is also logically a good solution. 

 ### Server components explained

-Message Broker (ActiveMQ) acts as a central point of the system, passing messages in-between components. ActiveMQ supports broker redundancy for improving scalability and reliability. 
-
-File server distributes files to other components, acting as a supplement to message broker. File distribution is based on pull mechanism, where components receiving data access file server and retrieve needed files. This way compute servers and clients can be behind firewalls. Using separate file servers also allows compute servers to use minimal disk space as files are cached at file server. 
-
-Authenticator processes requests from clients. Each request is examined, and if valid session exists for that client is is allowed to continue. Otherwise a request is made for user to authenticate and after a successfull authentication a new session is created. Authenticator supports many types of authentication sources (Unix passwd, JAAS, LDAP...), and can use them simultanously. Server components authenticate to broker using server specific keys, and are allowed to communicate directly without going through the authenticator. Authenticator is a separate component so that it can be deployed inside intranet, as it might need access to sensitive information such as user databases. 
-
-Computer server listens for computation requests. When client initiates a new task, all compute servers with free resources reply and client decides which server gets to process the task. This way there is no single point of failure in distribution of tasks to server environment, and compute servers can be modified easily on fly. Compute servers must have identical tool descriptions, but tools can be activated and inactivated per computer server, allowing different tool selections on different servers. 
+Message Broker (ActiveMQ) acts as a central point of the system, passing messages in-between components. ActiveMQ supports broker redundancy for improving scalability and reliability, so multiple brokers can be used simultaneously. 
+
+File broker distributes files to other components, acting as a supplement to message broker. File distribution is based on pull mechanism, where components needing data go and retrieve needed files from the file broker. This way compute servers and clients can be behind firewalls. Using separate file broker also allows compute servers to use minimal disk space as files are cached at file server. 
+
+Authenticator processes requests from clients. Each request is examined, and if valid session exists for that client it is allowed to continue. Otherwise a request is made for user to authenticate and after a successfull authentication a new session is created. Authenticator supports many types of authentication sources (Unix passwd, JAAS, LDAP...), and can use them simultanously. Server components authenticate to broker using server specific keys, and are allowed to communicate directly without going through the authenticator. Authenticator is a separate component so that it can be deployed inside intranet, as it might need access to sensitive information such as user databases. 
+
+Compute service listens for computation requests. When client initiates a new task, all compute services with free resources reply and client decides which service gets to process the task. This way there is no single point of failure in distribution of tasks to server environment, and compute services can be modified easily on fly. Compute services must have identical tool descriptions, but tools can be activated and inactivated per compute service, allowing different tool selections on different servers. 

 ### Simple server installation

-The simple way to install Chipster environment is to deploy all components to a single server and distribute clients either as standalone applications or by using Java Web Start. 
-
-All server components run inside their own directories, so having them on a single server does not require any special arrangements. Message broker and file server are running in their respective ports, and other components connect to them using local network loopback. 
+The simple way to install Chipster environment is to deploy all components to a single server and distribute clients by using Java Web Start. 
+
+All server components run inside their own directories, so having them on a single server does not require any special arrangements. Message broker and file broker are running in their respective ports, and other components connect to them using local network loopback. 

 ### Advanced server installation

-A good guideline for setting up advanced installation is to dedicate an untrusted server for message broker and file server components, as they are the only components that have open server ports. That server should not be inside organisations firewall ie. be in some type of DMZ network. To secure user credentials, authenticator should be installed separately on a strongly protected machine. 
-
-It is possible to deploy multiple compute servers. All of them should have same tools descriptions, but it is possible to select enabled tools per server. It is also possible to configure maximum job counts. If you have many nodes available but they have also other use besides Chipster it is recommended to deploy compute servers on as many nodes as possible but limit the per server job count to keep Chipster from hogging all the resources. If there are memory intensive tools, it might be a good idea to deploy dedicated node for them with large memory and low maximum job count. 
+A good guideline for setting up advanced installation is to dedicate an untrusted server for message broker and file broker components, as they are the only components that have open server ports. That server should not be inside organisations firewall, i.e., be in DMZ network. To secure user credentials, authenticator should be installed separately on a strongly protected machine. 
+
+It is possible to deploy multiple compute servers. All of them should have same tools descriptions, but it is possible to select enabled tools per server. It is also possible to configure maximum job counts. If you have many nodes available but they have also other use besides Chipster it is recommended to deploy compute servers on as many nodes as possible but limit the per server job count to keep Chipster from hogging all the resources. If there are memory intensive tools, it might be a good idea to deploy dedicated node for them with large memory and low maximum job count. Independent compute services can also be deployed to batch processing system (LSF etc.), following a worker paradigm. 

 ### Chipster and firewalls

-One of the design guidelines in Chipster was to make it easily adaptable to various firewall configurations. Even though there are many server components, only message broker and file server are listening to open ports. In other words, they act as a hub to which other components connect to. Both of the components are designed so that they can be installed on a "untrusted" machine located in the network's DMZ. Compute servers and authenticators often have to be located inside intranet, which is not a problem as they do not act as servers from network point of view. 
-
-Client uses TCP or SSL to connect to message broker and file server. This communication can be configured to ports 80 and 443 to bypass strict firewalls. In some high security environments practically all network access is disabled, except for HTTP using local proxy. Currently Chipster does not use HTTP, so in this extreme case deployment is not possible without changes to firewall configuration. However routing messages through HTTP is supported by ActiveMQ message broker, so in future these scenarios might also be supported directly. 
-
-  
-
-
-## Upgrading server installation to new Chipster version
-
-Here you can find short instructions on updating existing Chipster server installation with a newer version. The basic principle is that non backwards compatible changes are introduced between major versions (1.0, 1.1, etc.), so when upgrading between minor versions typically only Java libraries (JAR) need updating. Possible exceptions are documented here. 
+One of the design guidelines in Chipster was to make it easily adaptable to various firewall configurations. Even though there are many server components, only message and file brokers are listening to open ports. In other words, they act as a hub to which other components connect to. Both of the components are designed so that they can be installed on a "untrusted" machine located in the DMZ. Compute and authentication services often have to be located inside intranet, which is not a problem as they do not act as servers from a networking point of view. 
+
+Client uses TCP or SSL to connect to message and file brokers. This communication can be configured to ports 80 and 443 to bypass strict firewalls. In some high security environments practically all network access is disabled, except for HTTP using local proxy. Currently Chipster does not use HTTP, so in this extreme case deployment is not possible without changes to firewall configuration. However routing messages through HTTP is supported by ActiveMQ message broker, so in future these scenarios might also be supported directly. 
+
+  
+
+
+## Upgrading server installation
+
+Here you can find short instructions on updating existing Chipster server installation with a newer version. The basic principle is that non-backwards compatible changes are introduced between major versions (1.0, 1.1, etc.), so when upgrading between minor versions typically only Java libraries (JAR) need updating. Possible exceptions are documented here. 

 ### Upgrading from 1.4.x to later 1.4.x

@@ -387,13 +388,13 @@

 ### Configuring Chipster

-_If you just want to get your Chipster up and running, execute **configure.sh** script and your done! If you want to know more about Chipster configuration system, then read on._
-
-Chipster stores application configuration to a file called **chipster-config.xml**. It is located either in a conf subdirectory (see [directory layout](DirectoryLayout)) or loaded dynamically via URL. The former approach is meant for server components and the latter for clients starting over Java Web Start. The configuration file is not created automatically any more, but it most always exists (locally or behind a URL). 
-
-The configuration is loaded in two steps. First an internal default configuration is loaded (**chipster-config-specification.xml,** located inside the Chipster JAR) and then the normal configuration file **chipster-config.xml**. The latter contains only information that needs to be set per instance basis, or in other words, is quite minimalistic. However it is possible to overwrite configuration entries of the internal default configuration using the normal configuration file. Just include the entry in the file and it will replace the default one. 
-
-The recommended way to configure a new Chipster instance is to use the **configure.sh** script located at the installation root directory. It will configure all the components and the Web Start client descriptor. You can also modify the configuration files manually. For information on meaning of the different configuration entries, please refer to **[chipster-config-specification.xml](http://chipster.svn.sourceforge.net/viewvc/chipster/trunk/src/main/resources/chipster-config-specification.xml?view=markup)** in the code repository. 
+If you just want to get your Chipster up and running, execute **configure.sh** script and your done! If you want to know more about Chipster configuration system, then read on. 
+
+Chipster stores application configuration to a file called **chipster-config.xml**. It is located either in a conf subdirectory (see [directory layout](DirectoryLayout)) or loaded dynamically via URL. The former approach is meant for server components and the latter for clients starting over Java Web Start. The configuration file is not created automatically any more, but it must always exists (locally or behind an URL). 
+
+The configuration is loaded in two steps. First an internal default configuration is loaded (**chipster-config-specification.xml,** located inside the Chipster JAR) and then the normal configuration file **chipster-config.xml**. The latter contains only information that needs to be set per instance basis, so it is quite minimalistic. However it is possible to overwrite configuration entries of the internal default configuration using the normal configuration file. Just include the entry in the file and it will replace the default one. 
+
+The recommended way to configure a new Chipster instance is to use the **configure.sh** script located at the installation root directory. It will configure all the components and the Web Start client descriptor. You can also modify the configuration files manually. For information on meaning of the different configuration entries, please refer to **[http://code.google.com/p/chipster/sou.../chipster-config-specification.xml](http://code.google.com/p/chipster/source/browse/src/main/resources/chipster-config-specification.xml)** in the code repository. 

 ### Loading configuration over URL

@@ -401,7 +402,8 @@

 ### The configuration file

-The configuration file **chipster-config.xml** contains all the configuration that different components require. See below for an example configuration file of a filebroker component. 
+The configuration file **chipster-config.xml** contains all the configuration that different components require. See below for an example configuration file of a file broker component. 
+    

     <configuration content-version="3">

@@ -471,7 +473,7 @@

 **Step 1. Locate keystore**

-You can either use the [dummy keystore](http://chipster.svn.sourceforge.net/viewvc/chipster/trunk/src/main/resources/keystore.ks?view=log) that is bundled with Chipster clients and generate your own (see [GeneratingSslKeys]). Save it to file keystore.ks. 
+You can either use the [[keystore](http://code.google.com/p/chipster/source/browse/src/main/resources/keystore.ks|dummy)] that is bundled with Chipster clients and generate your own (see [#Generating_SSL_keys]). Save it to file keystore.ks. 

 **Step 2. Configure message broker**

@@ -490,11 +492,11 @@
   * copy keystore.ks to chipster/<component>/security 
   * open chipster/<component>/conf/chipster-config.xml and in module "messaging" change protocol to "ssl" (you can change port also) 

-That's it. You also need to change setting in the module "security" if you have used other than default values; see [GeneratingSslKeys](../../GeneratingSslKeys) for more details. 
+That's it. You also need to change setting in the module "security" if you have used other than default values; see [#Generating_SSL_keys] for more details. 

 ### Generating SSL keys

-Chipster comes with dummy keystore that gets you going with SSL. If you want to use SSL not only for encrypting communication but also establishing trust between server components and clients, you have to replace these publicly available keys with your own ones. Chipster uses Java's normal SSL implementation. Keystore can be manipulated as explained in [Java Security documentation](http://java.sun.com/security/reference/docs/i), so you can also use your existing keys. 
+Chipster comes with dummy keystore that gets you going with SSL. If you want to use SSL not only for encrypting communication but also establishing trust between server components and clients, you have to replace these publicly available keys with your own ones. Chipster uses Java's normal SSL implementation. Keystore can be manipulated as explained in [[Security documentation](http://java.sun.com/security/reference/docs/|Java)], so you can also use your existing keys. 

 Here we describe how you can generate your own SSL keys. Please note that these keys will not be approved by any Certificate Authority, and cause warnings if used outside of Chipster environment. 

@@ -574,9 +576,10 @@

 You should also follow [conventions for Chipster analysis tools](ChipsterToolConventions). 

-### Making R-scripts Chipster-compatible
-
-Chipster uses regular R-scripts. The only thing to remember is that interactive functions can not be used.  
+### Making R scripts Chipster compatible
+
+Chipster uses regular R scripts. The only thing to remember is that interactive functions can not be used. 
+
 Before running the script, the system runs the following initialisation snippet: 

     setwd(".")
@@ -587,9 +590,9 @@

 ### Writing VVSADL header

-R-scripts must be described by using a Chipster-specific description format called VVSADL (Very Very Simple Analysis Description Language). A VVSADL-snippet is added in comments as a header to a R-script file. The specification for the VVSADL can be found from the Javadoc documentation of the class fi.csc.microarray.VVSADLSyntax. Reasonably up-to-date copy of the document should be available from:  
-http://chipster.csc.fi/javadocs/latest/fi/csc/microarray/description/VVSADLSyntax.html 
-
+R-scripts must be described by using a Chipster specific description format called VVSADL (Very Very Simple Analysis Description Language). A VVSADL-snippet is added in comments as a header to a R-script file. The specification for the VVSADL can be found from the Javadoc documentation of the class [[http://chipster.csc.fi/javadocs/1.4/f.../VVSADLSyntax.html|fi.csc.microarray.VVSADLSyntax](http://chipster.csc.fi/javadocs/1.4/fi/csc/microarray/description/VVSADLSyntax.html|fi.csc.microarray.VVSADLSyntax)]. 
+
+  
 Here is an example of a VVSADL snippet to attach before of the actual R code: 

     # ANALYSIS Test/test (Just a test analysis for development)
@@ -608,11 +611,67 @@

 ### VVSADL input and output files

-Please see separate page [AnalysisToolInputsAndOutputs] for information. 
+Analysis tools inputs and outputs are files. Chipster system takes care that all input files are in place before the analysis tool is started and it is tool's responsibility to take care that all output files are produced when it ends. 
+
+Input and output definition line has format: 
+    
+    # INPUT <TYPE> files[...].tsv OUTPUT results.tsv
+
+Input files need to have a specified type. They can be either single files or a series of files, which are denoted with **[...]**. Output files do not have types and they are always single files. 
+
+It is possible to combine input and output definitions with comma, for example: 
+    
+    # INPUT <TYPE1> file.tsv, <TYPE2> files[...].tsv OUTPUT results1.tsv, results2.png, results3.mp3
+
+In this case, possible set of input files could be for example file.tsv, files1.tsv, files2.tsv and files3.tsv. Analysis tool would have to produce results1.tsv, results2.png and results3.mp3 always. 
+
+Supported input types are: 
+
+  * CDNA 
+    * Is a generic raw microarray dataset 
+    * Must not have a separate textual header (i.e., must begin with header row) 
+    * Must have column "sample" 
+  * AFFY 
+    * Is a Affymetrix CEL-file (text or binary) 
+    * Must have CEL-header 
+    * Must have extension .cel 
+  * GENE_EXPRS 
+    * Is a expression value file 
+    * Must not have a separate textual header (i.e., must begin with header row) 
+    * Must have column or columns with names starting with "chip." 
+  * GENELIST 
+    * Is a list of gene/probeset names 
+    * Must not have a separate textual header (i.e., must begin with header row) 
+    * Must contain column "identifier" or row name column with empty name (R style) 
+  * GENERIC 
+    * Can be any file 
+
+Definitions are inclusive, so tables can contain other columns besides the required ones (and usually do). 
+
+#### Input binding at client
+
+The analysis tool defines a set of input files. At the client side, when an operation is selected the client compares the set of selected datasets to what the operation specifies. So to speak in computer science terms, it does binding of concrete inputs to formal inputs. In a nutshell, formal inputs (as defined by the operation) are bound to concrete inputs (as chosen by user) using greedy and order-based algorithm. Formal inputs are processed in order and first fitting concrete input is bound. If formal input can have multiple concrete inputs (it is series of files), then all fitting ones are bound (greedy binding). Always at least one concrete input must be bound and a single concrete input cannot be bound multiple times. In the end all concrete inputs must be bound. 
+
+The one (and only) exception is phenodata. User does not need to select the phenodata dataset, but instead it is located automatically by looking at the relations between datasets. So phenodata is "inherited" when for example normalised dataset is filtered to produce a new dataset. It is possible for user to select the appropriate phenodata explicitly, but as it would be cumbersome in most cases, automatical deduction is offered. 
+
+When binding a formal input to a concrete dataset, client looks at the dataset and sees if it matches the type of the input. Basically it either looks at the file extension (.cel for example) or at the names of the table columns ("sample" for example). 
+
+#### Output post-processing at client
+
+Output files are connected to input files with a derivation type of link and placed to a proper folder. However phenodata file gets special treatment from client. When a script produces phenodata file, client recognises it and does a couple of special things: 
+
+  * creates annotation links to other results (the dotted line) 
+  * creates **original_name** column by looking at the input dataset names 
+    * not created if column already exists 
+  * creates **description** column by looking at the input dataset names 
+    * not created if column already exists 
+
+  
+

 ### VVSADL parameters

-Parameters allow user to tune behavior of an analysis tool. They are shown in the graphical parameter panel in the Chipster user interface and stored to variables when running a tool. 
+Parameters allow user to control the behavior of an analysis tool. They are shown in the graphical parameter panel in the Chipster user interface and stored to variables when running a tool. 

 Parameter definition format is: 

@@ -649,11 +708,11 @@
     * For selecting one column from the phenodata 
     * Behaves exactly like COLUMN_SEL, but uses phenodata as input dataset 

-Numeric parameters allow also minimum and maximum values to be set, by using keywords **FROM** and **TO after** the parameter type. All parameters allow a default value, which is given by using the keyword **DEFAULT**. The default value must be a valid value for the parameter. User interface implements validity checking in real time, so writing "one" to a **INTEGER** text box or "10" to a **INTEGER** text box with maximum 5 results in immediate error shown in the parameter panel side. 
-
-### Validating annotated R-scripts
-
-A validator is provided in the Chipster-distribution that allows you to check script syntax before trying to deploy it. You can trigger the validator with command line parameter rcheck, followed with script name. So, for example: 
+Numeric parameters allow also minimum and maximum values to be set, by using keywords **FROM** and **TO** after the parameter type. All parameters allow a default value, which is given by using the keyword **DEFAULT**. The default value must be a valid value for the parameter. User interface implements validity checking in real time, so writing "one" to a **INTEGER** text box or "10" to a **INTEGER** text box with maximum 5 results in immediate error shown in the parameter panel. 
+
+### Validating annotated R scripts
+
+A validator is provided in the Chipster distribution that allows you to check script syntax before trying to deploy it. You can trigger the validator with command line parameter **rcheck**, followed with script name. So, for example: 

     java -jar chipster-x.x.x.jar rcheck my_scripts/script.R

@@ -666,34 +725,36 @@
 Current limitations (version 1.2.3): 

   * overriding scripts changes the order of analysis tools in client (overridden scripts will be last) 
-  * in some environments updates to files are not detected immediately (restart will fix this, if you are in a hurry) 
   * when adding new scripts (not overriding existing), compute service must be rebooted for them to become visible 
   * it is your responsibility to make sure that custom-scripts are synchronised (or at least not in conflict) across multiple compute services 

 (adding new scripts was introduced in version 1.2.3.) 

-You can also change the header of the script with custom-scripts, including the name and category of the scripts. Categories are created on fly. 
+You can also change the header of the script with custom-scripts, including the name and category of the script. Categories are created on fly. 

 ## Customising tool selection

 It is possible to configure compute service instance to run only a given set of tools or all but a given set of tools. Using this mechanism different tools can be distributed to different services, for example to deploy some tools to a special platform or to run different versions of R/Bioconductor at the same time. 

-Tool (analysis operation) includes and excludes are configured in **nami-work-files/nami-config.xml**. Naturally only one of the two can be used at a time. 
-
-Here's an example of the syntax: 
-
-  
-
-    
+Tool (analysis operation) includes and excludes are configured in **comp/conf/chipster-config.xml**. Naturally only one of the two can be used at a time. 
+
+Here's an example that uses first service to run only Affymetrix normalisation tool and the second service all the rest. 
+
+The **comp/conf/chipster-config.xml** of the first service contains: 
+    
+    
+    <entry entryKey="includeOperations">
+    <value>/R/norm-affy.R</value>
+    </entry>
+    
+
+The **comp/conf/chipster-config.xml** of the second service contains: 

     <entry entryKey="excludeOperations">
     <value>/R/norm-affy.R</value>
     </entry>

-    
-
-If we have one compute service configure using the example config and other service with **<value>...</value>** moved to **includeOperations** and comments moved to **excludeOperations**, we have effectively created a two compute service deployment that runs Affymetrix normalisation tool on one service and all the rest on other.

TechnicalManual14 modified by Aleksi Kallio

Aleksi Kallio — Thu, 26 Jun 2014 06:42:39 -0000

--- v14
+++ v15
@@ -23,7 +23,7 @@
 Following software needs to be installed:

   * Java 1.5 or later 
-  * R, exactly right version (see [list of corresponding versions](ChipsterVsRVersions)) 
+  * correct version of R statistical environment (see [list of corresponding versions](ChipsterVsRVersions)) 

 The following tcp ports need to be open in the firewall: 

@@ -36,7 +36,7 @@

 Installation packages can be obtained from http://chipster.sourceforge.net/downloads.shtml. 

-After downloading extract the tar archive. It contains directory "chipster", where all components are in their own subdirectories. 
+After downloading extract the tar archive. It contains directory "chipster", where all components are in their own subdirectories. It can be placed anywhere, but usually **/opt/chipster** is used. 

 Downloading and extraction can be done easily on command line (we use version 1.1.2 here): 

@@ -46,11 +46,13 @@

 **2) Installing external tools**

-If you have installed R to default location /opt/chipster/tools/R-2.9.0, you can install the R libraries needed by Chipster with the setup tool. Run (as root if needed): 
+No external tools are needed to start the server environment, but for the microarray analysis tools to work, R and a collection of libraries are needed. You can skip this step if you just want to get the system running first. 
+
+If you have installed R to default location /opt/chipster/tools/R-2.9.0, you can install the R libraries needed by Chipster with the setup tool directly. Otherwise you have to update **comp/conf/environment.xml** first with the correct location of the R binary. Next run (as root if needed): 

     ./setup.sh

-For more information on setup tool see [Setup tool documentation](SetupTool). 
+For more information on setup tool see [Setup tool section](#Tool_installation_in_Linux). 

 Some tools require additional dependencies and fulfilling them is the most difficult part of the installation. However those steps can be skipped and done later, as only few of the analysis tool require additional dependencies to be met. The setup tool will print out instruction for carrying out the remaining installation steps for additional tools and databases. For more information on them see [InstallingExternalApplications].

TechnicalManual14 modified by Aleksi Kallio

Aleksi Kallio — Thu, 26 Jun 2014 06:42:39 -0000

--- v13
+++ v14
@@ -2,7 +2,7 @@

 # Technical manual for Chipster 1.4

-The manual covers Chipster platform version 1.4 and older. For the user manual, please see https://extras.csc.fi/biosciences/chipster-manual/. 
+The manual covers Chipster platform version 1.4 and older. It instructs in setting up your own Chipster server, adding your own tools into Chipster, and more. For the user manual, please see https://extras.csc.fi/biosciences/chipster-manual/. 

 ## Introduction

TechnicalManual14 modified by Aleksi Kallio

Aleksi Kallio — Thu, 26 Jun 2014 06:42:38 -0000

--- v12
+++ v13
@@ -8,13 +8,17 @@

 Chipster is a versatile data analysis platform with an intuitive graphical user interface. The version 1.4 of the platform has been mainly used for microarray and proteomics data. 

+In the basic setup, Chipster is a client-server system. Chipster server can be run on a single server computer or even a laptop. The Chipster server itself actually contains multiple independent services, so it can be scaled across a cluster of servers to distribute computational and data transfer load. 
+
+The system consists of  compute, authentication, management and logging services, and message and file brokers, which act as communication channels between the components. 
+
 # System installation

 ## System installation in Linux

 These are instructions for installation using the automatic tools provided in the installation package. 

-### 0) Requirements
+**0) Requirements**

 Following software needs to be installed: 

@@ -28,9 +32,7 @@
   * 8081 for webstart service (optional) 

-
-
-### 1) Downloading and extracting installation package
+**1) Downloading and extracting**

 Installation packages can be obtained from http://chipster.sourceforge.net/downloads.shtml. 

@@ -42,10 +44,7 @@
     wget http://chipster.csc.fi/downloads/chipster-1.1.2.tar
     tar xf chipster-1.1.2.tar

-  
-
-
-### 2) Installing analysis libraries and applications
+**2) Installing external tools**

 If you have installed R to default location /opt/chipster/tools/R-2.9.0, you can install the R libraries needed by Chipster with the setup tool. Run (as root if needed): 

@@ -55,7 +54,7 @@

 Some tools require additional dependencies and fulfilling them is the most difficult part of the installation. However those steps can be skipped and done later, as only few of the analysis tool require additional dependencies to be met. The setup tool will print out instruction for carrying out the remaining installation steps for additional tools and databases. For more information on them see [InstallingExternalApplications]. 

-### 3) Configuring Chipster services
+**3) Configuring Chipster services**

 To configure the Chipster services, run the following to scripts. Both scripts will ask for confirmation before writing changes to files. Defaults should be fine for a local installation. 

@@ -64,10 +63,7 @@

 configure.sh configures all the components, and genpasswd.sh generates secure passwords that server components use to authenticate each other. Please note that if you distribute Chipster to several machines you have to manually set passwords. 

-  
-
-
-### 4) Starting and stopping Chipster services
+**4) Starting and stopping service**s 

 To start all the Chipster services, run: 

@@ -76,9 +72,7 @@
 In addition to 'start', 'stop', 'restart', and 'status' can also be used. 

-
-
-### 5) Testing the installation with the client
+**5) Testing installation**

 To start the client locally (on the same machine as the services), run: 

@@ -92,7 +86,8 @@

 The default username/password is chipster/chipster. Users can be added by editing the userlist at auth/nami-work-files/users. Chipster also supports a host of other more advanced authentication providers. 

-### 6) Starting services at boot time
+  
+**6) Starting services at boot time**

 The steps needed for making services start at boot time are somewhat system dependent. In most Linux systems two steps are needed: 

@@ -122,7 +117,7 @@

 And of course we are constantly improving the support for external applications. Feedback and suggestions are always welcome. 

-### What "external applications" mean?
+### What do "tools" mean?

 By external applications we mean the computational environment needed to run Chipster compute service. Chipster itself is plain Java and does not have any dependencies to external application other than Java Runtime Environment. We do package Chipster with Tanuki Software's free Java Service Wrapper for convenience, but using the wrapper is not required. So, without the external applications in place your compute service will boot up, but will not be able to run successfully any actual analysis jobs. 

@@ -220,6 +215,22 @@

+
+## Client installation in Linux
+
+Client installs automatically with Java Web Start. 
+
+## Installation in Mac OS X
+
+Chipster client is fully Mac OS X compatible and supported on Mac platforms. It installs automatically with Java Web Start. 
+
+Chipster server has experimental support for Mac OS X from version 1.4.7. 
+
+## Installation in Windows
+
+Chipster client is fully Windows compatible and supported on Windows platforms. It installs automatically with Java Web Start. 
+
+Chipster server has experimental support for Windows. As the bioinformatics tool environment is Unix oriented, doing a complete installation in Windows will require significant efforts. 

 # System administration

@@ -479,7 +490,7 @@

 That's it. You also need to change setting in the module "security" if you have used other than default values; see [GeneratingSslKeys](../../GeneratingSslKeys) for more details. 

-### Generating you own SSL keys
+### Generating SSL keys

 Chipster comes with dummy keystore that gets you going with SSL. If you want to use SSL not only for encrypting communication but also establishing trust between server components and clients, you have to replace these publicly available keys with your own ones. Chipster uses Java's normal SSL implementation. Keystore can be manipulated as explained in [Java Security documentation](http://java.sun.com/security/reference/docs/i), so you can also use your existing keys. 

@@ -526,10 +537,23 @@
       </entry>

             ...
+    

 Default configuration does not have SSL specific settings, so you need to add those entries. You should update values for "keypass" and "keyalias" to reflect appropriate settings for each component. The key alias refers to the trusted key, not the private key. The alias of the private key needs not to be configured, but the key needs to be in the keystore anyway. You can also change keystore path if you don't wish to store the keystore inside the "security" directory. 

-## Authentication via LDAP
+## Authentication
+
+### Users file
+
+The simplest supported authentication mechanism is the users file in **auth/security/users**. The format is: 
+    
+    
+    <username>:<password>:<exp. date as YYYY-MM-DD>:comment
+    
+
+Only username and password are required. Blank lines and comment lines starting with **#** are allowed. 
+
+### LDAP

 See [Authentication via LDAP](LDAP). 

@@ -538,7 +562,7 @@

 # Tool development

-## Writing Chipster analysis scripts
+## Writing Chipster tools

 Basically, you have to do three things: 

@@ -648,7 +672,7 @@

 You can also change the header of the script with custom-scripts, including the name and category of the scripts. Categories are created on fly. 

-## Customising compute service tool selection
+## Customising tool selection

 It is possible to configure compute service instance to run only a given set of tools or all but a given set of tools. Using this mechanism different tools can be distributed to different services, for example to deploy some tools to a special platform or to run different versions of R/Bioconductor at the same time. 

@@ -675,7 +699,7 @@

-## Conventions for Chipster analysis tools
+## Tools conventions

 Here we describe different conventions that should be followed when developing or integrating tools to Chipster. The goal in Chipster is always to produce a coherent user experience, for which reason a lot of attention should be placed to harmonising the tools.