Recent changes to TechnicalManual

TechnicalManual modified by Aleksi Kallio

Aleksi Kallio — Fri, 11 Jul 2014 12:33:30 -0000

--- v30
+++ v31
@@ -1,8 +1,9 @@
-[TOC]
-
-# Technical manual for Chipster
-
-The manual covers Chipster platform version 2.3 and later. It instructs in setting up your own Chipster server, adding your own tools into Chipster, and more. For the user manual, please see http://chipster.csc.fi/manual/. 
+# Technical manual for Chipster 2.3 to 2.12
+
+> **Note! This is an unmaintained archive site.**
+> Wiki has been moved to Github and current documentation is available at [https://github.com/chipster/chipster/wiki](https://github.com/chipster/chipster/wiki)
+
+The manual covers Chipster platform version 2.3 to 2.12. It instructs in setting up your own Chipster server, adding your own tools into Chipster, and more. For the user manual, please see http://chipster.csc.fi/manual/.

 ## Introduction

TechnicalManual modified by Aleksi Kallio

Aleksi Kallio — Thu, 26 Jun 2014 06:41:50 -0000

--- v29
+++ v30
@@ -720,6 +720,25 @@

+## NAT
+
+Sometimes the server environment needs to be installed behind NAT (network address translation). This usually happens when the server environment connects to internal network, which is not visible to public internet. To make the system accessible externally, NAT host is added that directs traffic between internal and external networks. Nowadays such a setup if typical especially for cloud installation (e.g. OpenStack based environment). 
+
+Operating through NAT makes network configuration more complicated, because servers need to bind to different addresses that they should be connected to. Chipster supports operation through NAT, but needs some manual tweaks to configuration after running the automatic configuration script. 
+
+Required steps are: 
+
+  * Run configure.sh 
+    * Use internal address for message broker (ActiveMQ) host 
+    * Use external address for file broker host 
+  * Edit webstart/web-root/chipster.jnlp and change all occurrences of internal address to external address 
+  * Edit webstart/web-root/chipster-config.xml and change all occurrences of internal address to external address 
+
+Internal address is the actual IP or host name of the node that the server is running on. External address is the NAT'ed externally visible IP or host name that is mapped to the internal address. Same port numbers must be used internally and externally. 
+
+  
+
+
 ## Secure communications

 ### Setting up SSL
@@ -818,9 +837,6 @@
 ### LDAP

 See [Authentication via LDAP](LDAP). 
-
-  
-

 # Server components

TechnicalManual modified by Aleksi Kallio

Aleksi Kallio — Thu, 26 Jun 2014 06:41:49 -0000

--- v28
+++ v29
@@ -400,7 +400,7 @@

 **Up-to-date steps for installing all external applications and datasets can be found from the VM distribution installation script:**

-<http://code.google.com/p/chipster/source/browse/src/main/admin/vm/install-chipster.sh?name=default>
+<https://github.com/chipster/chipster/blob/master/src/main/admin/vm/install-chipster.sh>

 Follow steps onwards from _Install external applications and datasets_. 

@@ -628,7 +628,7 @@

 Configuration is loaded in two steps. First an internal default configuration is loaded (**chipster-config-specification.xml,** located inside the Chipster JAR) and then the normal configuration file **chipster-config.xml**. The latter contains only information that needs to be set per instance basis, so it is quite minimalistic. However it is possible to overwrite configuration entries of the internal default configuration using the normal configuration file. Just include the entry in the file and it will replace the default one. 

-The recommended way to configure a new Chipster instance is to use the **configure.sh** script located at the installation root directory. It will configure all the components and the Web Start client descriptor. You can also modify the configuration files manually. For information on meaning of the different configuration entries, please refer to **[chipster-config-specification.xml](http://code.google.com/p/chipster/source/browse/src/main/resources/chipster-config-specification.xml)** in the code repository. 
+The recommended way to configure a new Chipster instance is to use the **configure.sh** script located at the installation root directory. It will configure all the components and the Web Start client descriptor. You can also modify the configuration files manually. For information on meaning of the different configuration entries, please refer to **<https://github.com/chipster/chipster/blob/master/src/main/resources/chipster-config-specification.xml>** in the code repository. 

 ### Loading configuration over URL

@@ -728,7 +728,7 @@

 **Step 1. Locate keystore**

-You can either use the [dummy keystore](http://code.google.com/p/chipster/source/browse/src/main/resources/keystore.ks|) that is bundled with Chipster clients and generate your own (see [#Generating_SSL_keys]). Save it to file keystore.ks. 
+You can either use the keystore.ks dummy keystore that is bundled with Chipster clients and generate your own (see [#Generating_SSL_keys]). Save it to file keystore.ks. 

 **Step 2. Configure message broker**

TechnicalManual modified by Aleksi Kallio

Aleksi Kallio — Thu, 26 Jun 2014 06:41:49 -0000

TechnicalManual modified by Aleksi Kallio

Aleksi Kallio — Thu, 26 Jun 2014 06:41:48 -0000

--- v26
+++ v27
@@ -254,7 +254,7 @@

 If you fail to start the client, there is typically something wrong in the network settings. See next section on how to automatically reconfigure the network as a quick fix. 

-Before starting to actually use Chipster, it is highly recommended to update the installation to get latest bug fixes. See [Upgrading_server_installation]. 
+Before starting to actually use Chipster, it is highly recommended to update the installation to get latest bug fixes. See [#Upgrading_server_installation]. 

 ### Configuring Chipster

TechnicalManual modified by Aleksi Kallio

Aleksi Kallio — Thu, 26 Jun 2014 06:41:48 -0000

--- v25
+++ v26
@@ -254,7 +254,7 @@

 If you fail to start the client, there is typically something wrong in the network settings. See next section on how to automatically reconfigure the network as a quick fix. 

-Before starting to actually use Chipster, it is highly recommended to update the installation to get latest bug fixes. See [#Upgrading_server_installation]. 
+Before starting to actually use Chipster, it is highly recommended to update the installation to get latest bug fixes. See [Upgrading_server_installation]. 

 ### Configuring Chipster

TechnicalManual modified by Aleksi Kallio

Aleksi Kallio — Thu, 26 Jun 2014 06:41:47 -0000

--- v24
+++ v25
@@ -254,7 +254,7 @@

 If you fail to start the client, there is typically something wrong in the network settings. See next section on how to automatically reconfigure the network as a quick fix. 

-Before starting to actually use Chipster, it is highly recommended to update the installation to get latest bug fixes. 
+Before starting to actually use Chipster, it is highly recommended to update the installation to get latest bug fixes. See [#Upgrading_server_installation]. 

 ### Configuring Chipster

TechnicalManual modified by Aleksi Kallio

Aleksi Kallio — Thu, 26 Jun 2014 06:41:46 -0000

--- v23
+++ v24
@@ -100,7 +100,7 @@

 Note that the total size is around hundred gigabytes or more. 

-#### Add Chipster virtual
+#### Add Chipster virtual machine

 We are going to use bridged network. If you are using RedHat or Fedora Linux, you first need to disable NetworkManager, as it does not support bridged mode: 

@@ -251,6 +251,10 @@
 and clicking on the Launch Chipster link. Login with chipster/chipster. To get started, you can open an example session (link in Datasets panel). 

 If you don't know the hostname or IP address of the virtual machine you have started, see instructions in the next section. 
+
+If you fail to start the client, there is typically something wrong in the network settings. See next section on how to automatically reconfigure the network as a quick fix. 
+
+Before starting to actually use Chipster, it is highly recommended to update the installation to get latest bug fixes. 

 ### Configuring Chipster

TechnicalManual modified by Chipster admin

Chipster admin — Thu, 26 Jun 2014 06:41:46 -0000

--- v22
+++ v23
@@ -836,9 +836,18 @@

 File broker is required. There can be one or more file brokers in the system. When file brokers are added, clients and services start to distribute files randomly on the available file brokers. 

-### Restoring metadata database backup
+  
+
+
+### Metadata database backups

 Metadata database is part of file broker starting from Chipster 3.0. 
+
+By default metadata database is backed up 10 past midnight everyday and 100 backups are kept by deleting oldest backups as needed. This results in having daily backups for about 3 months time. 
+
+You can change metadata backup settings in `fileserver/conf/chipster-config.xml`. 
+
+#### Restoring metadata database backup

   * Stop chipster filebroker `service chipster-fileserver stop` or `/opt/chipster/fileserver/bin/linux-x86-64/chipster-fileserver stop`
   * Delete corrupted or old database `db-root/ChipsterFilebrokerMetadataDatabase`

TechnicalManual modified by Chipster admin

Chipster admin — Thu, 26 Jun 2014 06:41:45 -0000

--- v21
+++ v22
@@ -836,6 +836,17 @@

 File broker is required. There can be one or more file brokers in the system. When file brokers are added, clients and services start to distribute files randomly on the available file brokers. 

+### Restoring metadata database backup
+
+Metadata database is part of file broker starting from Chipster 3.0. 
+
+  * Stop chipster filebroker `service chipster-fileserver stop` or `/opt/chipster/fileserver/bin/linux-x86-64/chipster-fileserver stop`
+  * Delete corrupted or old database `db-root/ChipsterFilebrokerMetadataDatabase`
+  * Copy backed up database `metadata-backups/filebroker-metadata-db-backup-yyyy-mm-dd_hh-mm:ss/ChipsterFilebrokerMetadataDatabase` to `db-root`
+  * Restart chipster filebroker `service chipster-fileserver start` or `/opt/chipster/fileserver/bin/linux-x86-64/chipster-fileserver start`
+
+It is also possible to restore a backup by using certain connection URL attributes, but at the moment that requires changing the java code. See http://db.apache.org/derby/docs/10.9/adminguide/cadminhubbkup98797.html for more information. 
+
 ## Compute service

 Compute service takes care of all processing (tools in the client). It calls various backend applications and runtimes to do the actual computation.