Recent changes to High Performance Computing

High Performance Computing modified by Zhixiang Zhu

Zhixiang Zhu — Sun, 15 Sep 2013 15:32:39 -0000

High-performance computing modified by Zhixiang Zhu

Zhixiang Zhu — Sun, 15 Sep 2013 15:31:55 -0000

--- v2
+++ v3
@@ -2,8 +2,7 @@
 * * *
 This module generates a qsub script and submits it to the cluster if needed.

-**Subcommand**
-hpc
+**Subcommand: ** hpc

 **Available Options**
 --email

High-performance computing modified by Zhixiang Zhu

Zhixiang Zhu — Sun, 15 Sep 2013 15:31:03 -0000

--- v1
+++ v2
@@ -1,29 +1,28 @@
-**High-performance computing**
+**High Performance Computing**
 * * *
-This procedure provides an build-in option that helps submit the analysis to the cluster.
+This module generates a qsub script and submits it to the cluster if needed.

-**Master options**
---shell
-It only generates the shell script but not submits.
+**Subcommand**
+hpc

---qsub
-It generates a shell script and submits the analysis onto the cluster.
-
-**Options**
---email
+**Available Options**
+--email 
 Some cluster requires the user to specify the email for a submitted job.  This option can specify the email address.

---ram
-It specifies the memory required for the job.  The default memory is 10G.
+--ram 
+It specifies the memory required for the job.  The default memory is 10G. The argument must be an integer.

---name
+--name 
 It specifies the name of the job.  The default name of a job is gear.
+
+--submit
+If this option is set, the generated script will be submitted to the cluster.

 **Examples**
-gear --shell --name hpc-test --ram 80G --email guobo.chen@uq.edu.au --he-sd --grm test --pheno test.phe --mpheno 1 --out test
+gear hpc --name hpc-test --ram 80G --email guobo.chen@uq.edu.au --he-sd --grm test --pheno test.phe --mpheno 1 --out test

-gear --qsub --name hpc-test --ram 80G --email guobo.chen@uq.edu.au --he-sd --grm test --pheno test.phe --mpheno 1 --out test
+gear hpc --name hpc-test --ram 80G --email guobo.chen@uq.edu.au --submit --he-sd --grm test --pheno test.phe --mpheno 1 --out test

 The first example generates a shell script "hpc-test.sh", which reads below:
 >\#$ -cwd
@@ -31,7 +30,6 @@
 >\#$ -N HE_MAF_01
 >\#$ -m eas
 >\#$ -M guobo.chen@uq.edu.au
->java -jar -Xmx80G /clusterdata/gc5k/bin/gear.jar --name hpc-test --ram 80G --email guobo.chen@uq.edu.au --he-sd --grm test --pheno test.phe --mpheno 1 --out test
+>java -jar -Xmx80G /clusterdata/gc5k/bin/gear.jar --he-sd --grm test --pheno test.phe --mpheno 1 --out test

 The second example generates the shell script same as above and submits it onto the cluster.
-

Discussion for High-performance computing page

Zhixiang Zhu — Sun, 15 Sep 2013 09:33:22 -0000

I see in the code that the default value of --email is guobo.chen@uq.edu.au. I'll remove this default value; otherwise Guobo you will receive junk mails and will have to change the code once you change your frequently used mail address.

Discussion for High-performance computing page

Zhixiang Zhu — Sun, 15 Sep 2013 08:58:20 -0000

I don't know what the difference of QBI and Cheaha is, may QBI doesn't use qsub? But seems that an abstraction of different HPC platform is needed, if there're platform specific qsub options, or we want to support different HPC platforms.

Discussion for High-performance computing page

Guobo Chen — Sun, 15 Sep 2013 08:49:31 -0000

Another reason is that the queuing system may be quite different from one HPC to another.
For example, I configure the shell script only for the QBI cluster, which manages jobs different from Cheaha.

Discussion for High-performance computing page

Zhixiang Zhu — Sun, 15 Sep 2013 06:34:55 -0000

Why not just use qsub command directly instead?

WikiPage High-performance computing modified by Guobo Chen

Guobo Chen — Sun, 24 Mar 2013 23:19:29 -0000

High-performance computing

This procedure provides an build-in option that helps submit the analysis to the cluster.

Master options
--shell
It only generates the shell script but not submits.

--qsub
It generates a shell script and submits the analysis onto the cluster.

Options
--email
Some cluster requires the user to specify the email for a submitted job. This option can specify the email address.

--ram
It specifies the memory required for the job. The default memory is 10G.

--name
It specifies the name of the job. The default name of a job is gear.

Examples
gear --shell --name hpc-test --ram 80G --email guobo.chen@uq.edu.au --he-sd --grm test --pheno test.phe --mpheno 1 --out test

gear --qsub --name hpc-test --ram 80G --email guobo.chen@uq.edu.au --he-sd --grm test --pheno test.phe --mpheno 1 --out test

The first example generates a shell script "hpc-test.sh", which reads below:

#$ -cwd
#$ -l vf=80G
#$ -N HE_MAF_01
#$ -m eas
#$ -M guobo.chen@uq.edu.au
java -jar -Xmx80G /clusterdata/gc5k/bin/gear.jar --name hpc-test --ram 80G --email guobo.chen@uq.edu.au --he-sd --grm test --pheno test.phe --mpheno 1 --out test

The second example generates the shell script same as above and submits it onto the cluster.