Re: [scipion-users] PARALLEL_COMMAND in hosts.conf

[Grigory S. <sha...@gm...>]

Hi guys,

I think I found a workaround, the problem is that in my case for CPU relion
queues the

PARALLEL_COMMAND = mpiexec -mca orte_forward_job_control 1 -n 120 ...
and in template:
#$ -pe openmpi 5
#$ -l dedicated=24

where 24 is cores per node (default, to book only whole nodes), 5 is number
of nodes, 120 is total mpi*cores, threads = 1 (in relion GUI mpi=120,
threads=1)
----------
While for GPU queue:
PARALLEL_COMMAND = mpiexec -mca orte_forward_job_control 1 -n 5 ...
and in template:
#$ -pe openmpi 1
#$ -l dedicated=32

where 32 is cores per node(default, to book only whole nodes), 1 is number
of nodes, 5 is number of mpis (relion gui mpi=5, threads=6). I know it
looks weird, but that's how things are :)

Since %_(JOB_CORES)d (mpis * threads) can be used only in template but not
in parallel_command, I created a new var JOB_NODES2=mpi*threads/24, which
is overwritten in case of gpu. So here is my final config. I still have to
test it with non-relion programs though..

PARALLEL_COMMAND = mpiexec -mca orte_forward_job_control 1 -n
> %_(JOB_NODES)d %_(COMMAND)s
> SUBMIT_COMMAND = unset module; qsub %_(JOB_SCRIPT)s
> SUBMIT_TEMPLATE = #!/bin/sh
>        #$ -V
>        #$ -N scipion%_(JOB_NAME)s
>        #$ -pe %_(JOB_PE_TYPE)s %_(JOB_NODES2)d
>        #$ -l dedicated=%_(JOB_THREADS)d
>        #$ -e %_(JOB_SCRIPT)s.err
>        #$ -o %_(JOB_SCRIPT)s.out
>        #$ -cwd
>        #$ -S /bin/bash
>        %_(JOB_EXTRA_PARAMS)s
>        %_(JOB_COMMAND)s
> QUEUES = {
>        "CPU single node": [["JOB_PE_TYPE", "smp", "SGE PE type", "Select
> SGE PE type: openmpi or smp"], ["JOB_EXTRA_PARAMS", "", "Extra params",
> "Provide extra params for SGE"]],
>        "CPU 24 cores": [["JOB_PE_TYPE", "openmpi", "SGE PE type", "Select
> SGE PE type: openmpi or smp"], ["JOB_EXTRA_PARAMS", "", "Extra params",
> "Provide extra params for SGE"]],
>        "relion CPU 24 cores": [["JOB_PE_TYPE", "openmpi", "SGE PE type",
> "Select SGE PE type: openmpi or smp"], ["JOB_EXTRA_PARAMS", "#$ -l
> dedicated=24 -A Relion", "Extra params", "Use number of MPIs multiple of
> 24"]],
>        "GPU 32 cores": [["JOB_PE_TYPE", "openmpi", "SGE PE type", "Select
> SGE PE type: openmpi"], ["JOB_EXTRA_PARAMS", "#$ -pe openmpi 1 -l
> dedicated=32 -l gpu=4 -A Relion", "Extra params", "For Relion always use 5
> mpis, 6 threads"]]
>        }


To wrap up it would be good to have a bit more flexibility in hosts.conf


Best regards,
Grigory

------------------------------------------------------------
--------------------
Grigory Sharov, Ph.D.

MRC Laboratory of Molecular Biology,
Francis Crick Avenue,
Cambridge Biomedical Campus,
Cambridge CB2 0QH, UK.
tel. +44 (0) 1223 267542 <+44%201223%20267542>
e-mail: gs...@mr...

On Wed, Jun 28, 2017 at 10:52 AM, Jose Miguel de la Rosa Trevin <
del...@gm...> wrote:

> Hi Grigory,
>
> Can you comment a bit more about the situation? Why do you need to
> different PARALLEL_COMMAND in the same queue system?
> What I have seen (and used) it setting different variables in the
> SUBMIT_TEMPLATE that are defaulted to Void values for some
> queues and then defined in the specific ones (e.g., GPUs resource
> allocation when some nodes are just CPUs).
>
> Thanks for reporting,
> Bests,
> Jose Miguel
>
>
> On Wed, Jun 28, 2017 at 11:48 AM, Pablo Conesa <pc...@cn...>
> wrote:
>
>> Hi Grigory, I don't see a workaround, I'll add it to github. Thanks.
>>
>> On 27/06/17 18:43, Grigory Sharov wrote:
>>
>> Hi all,
>>
>> is it possible to provide a different PARALLEL_COMMAND in hosts.conf for
>> a specific queue? Would be also great to have multiple SUBMIT_TEMPLATE...
>>
>> Best regards,
>> Grigory
>>
>> ------------------------------------------------------------
>> --------------------
>> Grigory Sharov, Ph.D.
>>
>> MRC Laboratory of Molecular Biology,
>> Francis Crick Avenue,
>> Cambridge Biomedical Campus,
>> Cambridge CB2 0QH, UK.
>> tel. +44 (0) 1223 267542 <+44%201223%20267542>
>> e-mail: gs...@mr...
>>
>>
>> ------------------------------------------------------------------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>
>>
>>
>> _______________________________________________
>> scipion-users mailing lis...@li...https://lists.sourceforge.net/lists/listinfo/scipion-users
>>
>>
>>
>> ------------------------------------------------------------
>> ------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>> _______________________________________________
>> scipion-users mailing list
>> sci...@li...
>> https://lists.sourceforge.net/lists/listinfo/scipion-users
>>
>>
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> scipion-users mailing list
> sci...@li...
> https://lists.sourceforge.net/lists/listinfo/scipion-users
>
>