buildtemplateparallel with condor_q

Help
Arman
2014-01-11
2014-03-14
  • Arman

    Arman - 2014-01-11

    Dear all,

    I'm trying to make a study specific template with buildtemplateparallel.sh.
    The problem is that we use HTCondor as our grid engine, and according to my
    limited knowledge, ANTs seems not to be able to support Condor. I noticed
    it does support SGE so I made a symbolic link in /usr/bin using qsub that
    points toward condor_qsub. Seems that the method does not work and when the
    script tries to submit jobs I get the following error:

    condor_qsub: unrecognized option '-n'

    What I did was to went through buildtemplateparallel.sh to find '-n' option
    to somehow comment out the part that adds it to the final script
    but apparently couldn't find it and got more confused. Any inputs on using
    ANTs with Condor?

    All the best,
    Arman

    p.s. FSL does work with the trick above, using condor_qsub with a symbolic
    link.

     
  • Nick Tustison

    Nick Tustison - 2014-01-11

    SGE was the only grid engine used originally in the buildtemplateparallel script. However, when I moved to the University of Virginia, I modified that code and wrote antsMultivariateTemplateConstruction.sh to use PBS qsub. So you have two examples which use different qsub options to adapt the code yourself to Condor. Parallelization is only performed in two places: one for the rigid registration (if requested) and one for the deformable registration (per iteration) so you have adapt your qsub call in those two places. Additionally, you have to write a perl script to continually poll for when the jobs have been completed. See waitForPBSQJobs.pl and waitForSGEQJobs.pl for examples.

    Nick

     
    • Arman

      Arman - 2014-01-12

      I've forked the repo on github. Will add the functionality to work with
      Condor and will send you a pull request. Seems that Brian is the
      maintainer.

      All the best,
      Arman

      On Sun, Jan 12, 2014 at 12:58 AM, Nick Tustison ntustison@users.sf.netwrote:

      SGE was the only grid engine used originally in the buildtemplateparallel
      script. However, when I moved to the University of Virginia, I modified
      that code and wrote antsMultivariateTemplateConstruction.sh to use PBS
      qsub. So you have two examples which use different qsub options to adapt
      the code yourself to Condor. Parallelization is only performed in two
      places: one for the rigid registration (if requested) and one for the
      deformable registration (per iteration) so you have adapt your qsub call in
      those two places. Additionally, you have to write a perl script to
      continually poll for when the jobs have been completed. See
      waitForPBSQJobs.pl and waitForSGEQJobs.pl for examples.

      Nick

      buildtemplateparallel with condor_qhttps://sourceforge.net/p/advants/discussion/840261/thread/39c66e99/?limit=25#579f

      Sent from sourceforge.net because you indicated interest in
      https://sourceforge.net/p/advants/discussion/840261/

      To unsubscribe from further messages, please visit
      https://sourceforge.net/auth/subscriptions/

       
  • SG Kim

    SG Kim - 2014-02-08

    I don't know if you already have your own codes, Arman, but from my experience, it's quite simple as Nick said. All you need to do is just to add some conditional lines for condor at the ants registration loop, and to write a tiny script that checks jobs' cueing IDs periodically.

    For anyone who wants to check my modification: https://sourceforge.net/projects/ants-condor/

     
    Last edit: SG Kim 2014-02-08
  • Nick Tustison

    Nick Tustison - 2014-02-08

    Thanks for posting this.

     
    • Arman

      Arman - 2014-02-08

      I tried to change SGE submission to condor submission at the time with this
      (similar to line 969, buildtemplateparallel.sh):

      if [ $DOQSUB -eq 5 ] ; then
      id=*condor_qsub* -cwd -S /bin/bash -N antsBuildTemplate_rigid -v ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS=1,LD_LIBRARY_PATH=$LD_LIBRARY_PATH,ANTSPATH=$ANTSPATH $QSUBOPTS $qscript | awk '{print $3}'
      jobIDs="$jobIDs $id"

      But it never worked. Condor was complaining of extra -q option and I
      couldn't figure out were -q was coming from.

      Thanks for this. Maybe merging at github instead of sourceforge?

      On Sat, Feb 8, 2014 at 8:25 PM, Nick Tustison ntustison@users.sf.netwrote:

      Thanks for posting this.

      buildtemplateparallel with condor_qhttps://sourceforge.net/p/advants/discussion/840261/thread/39c66e99/?limit=25#d8c2

      Sent from sourceforge.net because you indicated interest in
      https://sourceforge.net/p/advants/discussion/840261/

      To unsubscribe from further messages, please visit
      https://sourceforge.net/auth/subscriptions/

       

Log in to post a comment.

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:





No, thanks