Bart Hazes - 2013-07-28

CDSbank was developed with python2.7 on a Ubuntu 11.10 GNU/Linux x86_64 server. Installation instructions below are for Ubuntu and related operating systems but should be useful for other platforms as well. Scripts and code should transfer with few/any problems to other Unix variants, including Mac OS X. Windows or other platforms without bourne-shell support will need more work.

I am currently going through the entire installation procedure on a ubuntu virtual machine and building up the instructions below as I go. Things will be changing in the next little while to make things more fool-proof and maintenance friendly. If you go ahead with installing the server please get in contact as I would like to hear about problems, successes, or suggestions. If you install on a different system that would also be good to know.

Prerequisites
1) Install mongodb (status: tested and working)
For more information about mongodb go to http://www.mongodb.org/
Here you can find documention and download mongodb binaries for many platforms including Windows, Mac Os X, Solaris, and Linux
For Ubuntu you can also use the 10gen repository as follows (note: each command must be one line):

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv 7F0CEB10

echo 'deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen' | sudo tee /etc/apt/sources.list.d/10gen.list

sudo apt-get update

sudo apt-get install mongodb-10gen

This installs mongodb and starts the mongod daemon. To try it out type mongo on the command line to establish an interactive connection to the database. On a fresh install it sometimes fails on first try with a "could not connect to server" message. Try again. Once in, at the prompt type: show dbs
This lists the default database(s). Just "local" in a fresh install. You now have a working mongodb install. Type <control-d></control-d> to exit.

Optionally: by default mongo runs as the mongodb user with files and logs directed to /var/lib/mongodb. You can change the /etc/mongodb.conf configuration file if needed. You can also set remote access and authorization for the database.

2) Install pymongo (status: tested and working)
This python module provides access to mongodb. The preferred method to install it is to type: sudo pip install pymongo

If you don't have pip on your system you can get it with:
sudo apt-get install python-pip python-dev build-essential

Alternatively you can try: sudo easy_install pymongo

3) Install ascp (status: tested and working)
The Aspera ascp program provides much faster NCBI data downloads. See http://www.ncbi.nlm.nih.gov/books/NBK47527/ for installation instructions.
(In short you must install the Aspera connect client from here http://www.asperasoft.com/downloads
Untarring the tar.gz file creates a shell script that will install the software when executed. In my install, the executables ended up in ~/.aspera/connect/bin and that is where the scripts expect it to be.

Extras
Mac OS X does not come with make pre-installed but it is available for free download Apple

The CDSbank server runs on an Apache webserver. Other servers should work as long as they allow cgi scripts/programs to run on the server.

4) CDSbank database building (status: tested and working)

4a) General set-up
Decide on a directory for all CDSbank files. The large Genbank data files will be stored in subdirectories so make sure you have enough space (my current setup takes almost 300MB but you want more to allow for growth). The mongo database, which has a similar size, can reside elsewhere (as defined in the /etc/mongo.conf file).

Fetch the latest CDSbank tarball from SourceForge and place it in the directory where you want to install CDSbank. Unpacking with tar xvzf CDSbank-#.#.#alpha.tgz will create a new directory with the following content:

GB2mongo.py
GBmongo.py
genbank.py
geneticCode.py
getVersions.sh
Makefile
mongoSettings.py
README
seqlib.py
Taxonomy2Mongo.py
updateDB.sh
cgi-bin/
html/

In an editor change the first line of the Makefile to indicate the installation directory for CDSbank. On my system is looks like this

export CDSBANK:=/data2/db/CDSbank

No other changes should be needed.

From the command line type: ./getVersions.sh

If your are connected to the network this should return the genbank and refseq release numbers. Currently it shows:
genbank197_refseq60 for genbank release 197 and refseq release 60. I recommend that you create a directory with this name to download the files and build the database. Note: the name of the directory will become the name of the database and you should change to this directory before performing any build/update/... actions.

mkdir genbank197_refseq60
cd genbank197_refseq60

From the command line type: make -f ../Makefile
The result is a list of BUILD targets with brief explanation

make BUILD Build entire CDSbank database
make CHECK Confirm correct build/update
make UPDATE Apply new daily updates
make INDEX Create extra indices
make ACTIVE Make CDSbank use this as the 'active' database
make CLEAN Delete all data files (can still update)
make REALCLEAN Delete data/log files (no more updates)
make DROPDB Delete the database (CHECK you're in the correct directory!!!)

To build the full database simply type:
make -f ../Makefile BUILD

This will first fetch and process the NCBI taxonomy database, followed by the WGS, GENBANK, and REFSEQ databases. On my system (6-core Intel core i7 3930K, using 6 threads) this takes about 43 hours and consumes 296 GB of disk space.

Next type: make -f ../Makefile CHECK

Upon success this returns
13-08-19/12:11:33 PM: CDSBANK updated. No unprocessed files left

If there are problems it will say that there are Some non-empty ToBeUpdated files left
If this happens you can find the offending file by typing:
find -name '.partial'
The last lines in this file may indicate the nature of the problem. If it appeared to have been a network interruption, you can just reissue the make -f ../Makefile BUILD* command and it will continue from where it left off, while reloading the problematic files.

If the build was successful you should apply all daily updates that have appeared since the major release. Do this by typing:
make -f ../Makefile UPDATE

You can optionally use the CHECK target again to confirm success.

During building a minimum number of indices are maintained to speed building. For full function, CDSbank requires additional indices which are made by typing:
make -f ../Makefile INDEX

On my system this takes about an hour. This only has to be done once, after the full build and initial update is complete.

The database is now complete and can be used. To do this its status has to be set to "active". This is done by typing:
make -f ../Makefile ACTIVE

If needed you can conserve disk space by deleting all genbank data files by typing:
make -f ../Makefile CLEAN

This step leaves ".Finished" files in place, which are used to indicate that this file has been successfully processed.

If a new database has been built you can clean up the old one by typing:
make -f ../Makefile DROPDB
make -f ../Makefile REALCLEAN

This removes both the mongo database and all genbank and ".Finished" files. For an even more rigorous clean you can replace the latter REALCLEAN action by a rm -f oldDBname command from the unix command line.

To perform daily updates I added the following line to the /etc/crontab file

00 6 * * * www cd /data2/db/CDSbank/mongoSettings.py --get_active; rm -f taxdump.tar.gz; make -f ../Makefile UPDATE >> CRONTAB; make -f ../Makefile CHECK >> UPDATE.log

Every morning at 6 AM this deletes the old taxonomy database, followed by a rebuild of the taxonomy database and all daily updates to the WGS, GENBANK and REFSEQ data sets. To make this work for you, you need to change /data2/db/CDSbank/ to the root location for CDSbank on your system.

5) The CDSbank user interface (status: not yet tested)
The distribution includes cgi-bin and html directories to replicate the CDSbank webservice on a local server. The html directory should be copied to a place where your webserver looks for html pages. Similarly, the cgi-bin directory should be copied to a place where your webserver looks for scripts/programs.
If you wish to use CDSbank as an on-line tool you can simply replace the code reading input from the web form at the top of the code with command line parser such as argparse. I will likely implement this myself at some point.

 

Last edit: Bart Hazes 2013-08-19