Re: [Opaltoolkit-users] Sequential Execution

[nadya w. <na...@sd...>]

Hello Jeff,


On Apr 9, 2012, at 10:29 AM, Jeffrey Allen wrote:

> Hello,
>
>
> My group has been working on developing an execution pipeline for  
> Bioinformatics programs. We’ve been considering using SOAP to  
> handle the input parameters for the individual programs; it seems  
> that Opal could be a good foundation for us to use if we go this  
> direction.
>
this is exactly how we use opal for individual programs.
>
> My question, however, is whether or not there are any built-in or  
> external projects that offer some control of the “flow” of  
> execution of submitted commands. For instance, a user may submit  
> requests A, B, and C, but would like to specify that B and C depend  
> on the output of A. Thus, the proper execution would be to begin A  
> immediately, then begin B and C once A has completed.
>
There are a few that might fit here. There are a few workflow engines  
and all have their pros and cons.
Take a look at
	kepler https://kepler-project.org/
	vision (part of mgltools) http://mgltools.scripps.edu/
	CADD - our first experimental workflows pipeline, http:// 
www2.nbcr.net/wordpress2/?page_id=1175
We had some experience with both. Both a are mainly gui-driven and  
provide one with the
capability to build  'individual nodes' and chain them into the  
pipeline. An individual node then
can be a SOPA call to a specific opal service.  The mgltools 1.5.6 r2  
distro has our experimental first cut
pipeline  with a few workflows.

In all these as far as opal is concerned, opal executes the job once  
it receives it  and it has no "knowledge"
of the jobs dependencies. What we did in situations like this, was to  
build an opal web service that takes
all the needed input  from the user and on the back end runs a script  
that runs certain portions in whatever order is
needed.  In this scenario one is building on top of opal. Each  
scripted pipeline then need to be carefully
crafted. Take a look at http://kryptonite.nbcr.net/opal2/dashboard? 
command=serviceList for AutoDock Vina Screening service,
it does exactly that. The SMAP services are another example.

Another option is to use condor DAG capability. One will need to use  
condor as a job scheduling platform and write
submit scripts that tell condor when and what to execute. This is  
outside opal.

>
> I’d be interested to hear any advice this group has on how to  
> proceed with a project like this. Is this something which could be  
> built on top of Opal, should I consider trying to incorporate this  
> directly into Opal, or is there another platform which I should  
> consider using?
>
to incorporate directly into opal means re-writing opal. Opal was not  
conceived as a scheduling manager. It prepares
the job for the execution and submits it to a real job scheduler like  
condor, pbs, sge. The submission script has to handle
creating all the right dependencies in a syntax that a real job  
scheduler understands.

Nadya

>
> Thanks,
>
>
> Jeff Allen
>
> Computational Biologist
>
> Quantitative Biomedical Research Center
>
> UT Southwestern Medical Center
>
> 214.648.4171 - NC8.512B
>
> Jef...@UT...
>
>
>
>
> UT Southwestern Medical Center
> The future of medicine, today.
> ---------------------------------------------------------------------- 
> --------
> Better than sec? Nothing is better than sec when it comes to
> monitoring Big Data applications. Try Boundary one-second
> resolution app monitoring today. Free.
> http://p.sf.net/sfu/Boundary- 
> dev2dev_______________________________________________
> Opaltoolkit-users mailing list
> Opa...@li...
> https://lists.sourceforge.net/lists/listinfo/opaltoolkit-users

Nadya Williams                 University of California, San Diego
na...@sd...             9500 Gilman Dr. MC 0446
+1 858 534 1820 (ofc)      La Jolla, CA 92093-0446
+1 858 822 1619 (fax)      USA