Re: [Opaltoolkit-users] Opal job scheduler benefits

[Guillermo M. P. <gui...@si...>]

Hello Nadya,

Thank you for the information. This starts to make sense.
I had no idea on how to pass Opal my Condor config.

I'm going to try to make this work with Grid universe. I'll report asap.


Once again, thank you very much.

Best regards,
Guillermo.

On 02/28/2013 06:32 PM, nadya williams wrote:
> Hi Guillermo,
>
> there are a few issues here.
> On Feb 28, 2013, at 12:47 AM, Guillermo Marco Puche wrote:
>
>> Hello Luca,
>>
>> I currently can't run Opal jobs with Condor job scheduler:
>>
>> Here's my basic app: bwa.xml --> http://pastebin.com/uGqvvBki
>> I know it has empty parameters but it's for testing purposes at this 
>> moment. All parameters and flags are run in a shell script invoked by 
>> condor job file.
> your xml file is not correct. You are using your condor.submit file 
>  in place of an executable:
> <binaryLocation>/opt/web/opal_scripts/bwa/bwa.condor</binaryLocation>
> Instead, here you need to use your /opt/web/opal_scripts/bwa/bwa.sh
> and any other parameters from which opal will make condor_submit file. 
>  Opal v 2.5 has condor.expr.file
> variable in opal.properties file:
> # Enable if there are server-specific condor expressions. Put 
> expressions in the file
> #condor.expr.file=/opt/opal/etc/condor.expr
>
> This is a file  (condor submit syntax) that needs to be used to tell 
> opal to add extra parameters  to every submit file that is generated.
> This allows for server-side specific variables to be added.
>
>>
>> I want opal to execute my bwa.condor file (condor job file) which 
>> currently works with condor_submit command but not with Opal.
> this is not going to happen using your current xml file. The way 
> condor+opal work is opal generates condor submit file
> from the parameters given in xml file. You have a universe that we did 
> not test with condor before.
> Currently, opal+condor is working with  vanila or parallel universe only.
>
> You can use a workaround:
>
> I suggest you try to make a wrapper script and use it in 
> <binaryLocation> in xml file.
> In you xml file add properties like
> (1) condor submit file (untagged parameter for upload of  the submit 
> file) as
>           <param>
>                 <id>submitFile</id>
>                 <paramType>FILE</paramType>
>                 <ioType>INPUT</ioType>
>                 <required>true</required>
>                 <textDesc>upload a condor submit file </textDesc>
>             </param>
>
> (2)  use  Fork Job Manager   as 
>  <jobManagerFQCN>edu.sdsc.nbcr.opal.manager.ForkJobManager</jobManagerFQCN>
> (3)  use  <parallel>false</parallel>
>
> please see apbs_parallel_1.3.xml  in the opal distro for an example.
> Your wrapper script need to understand that it supposed to look for a 
> submit file (check for presence)
>  and then  condor submission via "condor_submit yourfile"
>
> This way you will have a flexibility to create "any"  submit file and 
> use any universe  and other specifics of the
> condor submission that we currently don't handle for simple cases.
>
> Regards,
> Nadya
>
>
>>
>> Here's the content of bwa.condor: http://pastebin.com/P9GNwriJ
>> And here the bwa.sh invoked by bwa.condor to be executed on remote 
>> cluster: http://pastebin.com/dWzQa92E
>>
>> Best regards,
>> Guillermo.
>>
>>
>> On 02/27/2013 11:17 PM, Luca Clementi wrote:
>>> On Wed, Feb 27, 2013 at 4:46 AM, Guillermo Marco Puche
>>> <gui...@si...>  wrote:
>>>> Hello,
>>>>
>>>> I would like to know what are the benefits/extras of using Condor
>>>> job.scheduler with Opal.
>>> The jobs you submit to Opal will be executed using Condor.
>>> Opal simply gives you a web service interface and then it has
>>> different back-end to actually execute your jobs (condor, sge, pbs,
>>> etc.).
>>>
>>>> What's the difference between job scheduler and submitting jobs to Condor
>>>> straight with condor_submit?
>>> you mean the difference between using Opal vs using condor_submit?
>>> If you use Opal you can invoke the lanuchJob operation using
>>> web-service standard (we provide python and java client side
>>> libraries).
>>> If you use condor you have to ssh to a machine (aka have an account)
>>> and then you need to create a submission script and execute
>>> condor_submit.
>>>
>>>
>>> In NBCR We use Opal to submit job from web portal (where you have 1
>>> user the web portal which is in charge of running different simulation
>>> and opal takes care of creating working directories staging input and
>>> outputs etc.).
>>>
>>>
>>> Luca
>>
>>
>> -- 
>> <1MjpCpe.png> <http://i.imgur.com/1MjpCpe.png> *g.marco*: 
>> Informatician at Sistemas Genómicos S.L <x-msg://69/#>
>> phone: 0034635197460 <callto:0034635197460>
>> web: www.sistemasgenomicos.com <http://www.sistemasgenomicos.com/>
>> ------------------------------------------------------------------------------
>> Everyone hates slow websites. So do we.
>> Make your web apps faster with AppDynamics
>> Download AppDynamics Lite for free today:
>> http://p.sf.net/sfu/appdyn_d2d_feb_______________________________________________
>> Opaltoolkit-users mailing list
>> Opa...@li...
>> https://lists.sourceforge.net/lists/listinfo/opaltoolkit-users
>
> Nadya Williams  University of California, San Diego
> na...@sd... <mailto:na...@sd...>             9500 Gilman Dr. MC 0446
> +1 858 534 1820 (ofc)      La Jolla, CA 92093-0446
> +1 858 822 1619 (fax)      USA
>
>
>