#131 runCA for LSF

Assembly_pipeline
open
Jason Miller
Utility (7)
5
2011-11-18
2011-11-18
G. Koscielny
No

Hello,

I work at the EBI where our computing pipelines rely extensively on LSF.

Thus, I've adapted the runCA script and called it runCA_LSF to run it on our computing farm to do some transcriptome assembly and optimize the memory and thread usage.

From my recent test on a 454 assembly (> 2M reads), it works well until the terminator step where I had probably a buffer overflow issue.

So, after recompiling AS_MSG_pmesg.c with a bigger msgMax, I'll see if I can complete the pipeline using LSF.

It would be useful if you could distribute my script with the wgs-assembler. I've adapted also the spec file so it does not mix with SGE commands.

Example:

# See http://sourceforge.net/apps/mediawiki/wgs-assembler/index.php?title=RunCA_and_Sun_Grid_Engine
# By default, do NOT use LSF. The use must manually override these with command-line options.
#
useGrid = 1
scriptOnGrid = 1

# Once LSF is enabled, also do fragment and overlap correction under LSF control.
# By default, overlap and consensus jobs are under LSF control.
#
ovlOnGrid = 1
frgCorrOnGrid = 1
ovlCorrOnGrid = 1

lsf = -q production
lsfScript = -n 1 -M 8192 -R "rusage[mem=8192]"
lsfOverlap = -n 2 -M 16384 -R "rusage[mem=16384]"
lsfMerOverlapSeed = -n 2 -M 16384 -R "rusage[mem=16384]"
lsfMerOverlapExtend = -n 2 -M 16384 -R "rusage[mem=16384]"
lsfConsensus = -n 1 -M 8192 -R "rusage[mem=8192]"
lsfFragmentCorrection = -n 2 -M 32000 -R "rusage[mem=32000]"
lsfOverlapCorrection = -n 1 -M 32000 -R "rusage[mem=32000]"

etc.

Yours sincerely,

Gautier

Discussion

  • Jason Miller
    Jason Miller
    2011-11-18

    We are occasionally asked about support for LSF. Thus, I would say this is potentially useful. I would like to consider including in the upcoming CA 7 release. You posted this anonymously so you won't receive this reply by email. Would you contact us directly by email (atgATjcviDOTorg) or open a duplicate case using your Source Forge login?

     
  • Jason Miller
    Jason Miller
    2011-11-18

    • assigned_to: nobody --> jasonmiller9704
     
  • G. Koscielny
    G. Koscielny
    2011-11-18

    Hello,
    I did use my login to write this message and receive your comment in my mailbox.
    I'll write directly to the e-mail address you sent.

     
  • Jason Miller
    Jason Miller
    2011-11-18

    Thanks. Sorry I thought this was an anonymous post. I don't know why. -- Jason