Menu
▾
▴
pipeline130-developers — Developer Mailing List
You can subscribe to this list here.
| 2009 |
Jan
|
Feb
|
Mar
|
Apr
(37) |
May
(113) |
Jun
(62) |
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
|---|
|
From: <cs...@uc...> - 2009-05-21 07:13:22
|
By invalid workflows I mean any workflow that can be proven incorrect before being passed to the engine. So yes, cyclic graphs (we need multiple examples of varying complexity) but also graphs with identically-named jobs, or unnamed jobs, or just otherwise broken. Yes Mike, that's more or less what we want in terms of testing. But by "someone" I meant not you, me, or Nils. The wiki hasn't been updated since the group 8 review and the three of us are the only ones with more than one commit. -Chris Quoting Nils Homer <nil...@uc...>: > Use DFS to find cycles; it's O(N) and it's easy to implement. > > Chris: what do you mean by invalid workflows? Do you mean cyclic graphs? > Or do you mean workflows that fail (see the examples dir for some of these)? > > Nils > > On 5/20/09 11:19 PM, "Chris Swan" <cs...@uc...> wrote: > >> If that doesn't work, look into using just a depth-first search... pretty >> sure you can make it O(n) if you do it right and shouldn't be too hard to >> implement. >> >> Someone: come up with good test cases for invalid workflows! >> >> -Chris >> >> -------------------------------------------------- >> From: "Mike Hess" <mh...@uc...> >> Sent: Wednesday, May 20, 2009 11:05 PM >> To: <pip...@li...> >> Subject: Re: [Pipeline130-developers] Validating Workflows >> >>> So basically the idea is the number of vertices is the number of jobs >>> in the workflow, the number of edges is the sum of the number of >>> children each job has, then the number of components is the hard part, >>> but I added a pseudo-recursive function to check if is a new >>> component. I am taking a math graph class currently alongside the cs >>> class and I got the formula from the class (just took a test on it >>> haha.) I'm not 100% sure the formula works both ways but I'll look >>> into it deeper. >>> >>> >>> -Mike >>> >>> On May 20, 2009, at 10:54 PM, Mike Hess wrote: >>> >>>> vertices = edges + components for an acyclic graph I'm pretty sure. >>>> Should that work ok? Cuz I'm basically done with it right now, just >>>> need to add comments and test it a little. >>>> >>>> -Mike >>>> >>>> On May 20, 2009, at 10:42 PM, Chris Swan wrote: >>>> >>>>> As long as the graph is acyclic, it's OK -- the unconnected parts >>>>> will just >>>>> run in parallel. What algorithm are you planning on using to detect >>>>> cycles? >>>>> >>>>> -Chris >>>>> >>>>> -------------------------------------------------- >>>>> From: "Mike Hess" <mh...@uc...> >>>>> Sent: Wednesday, May 20, 2009 9:50 PM >>>>> To: "CS 130" <pip...@li...> >>>>> Subject: [Pipeline130-developers] Validating Workflows >>>>> >>>>>> I was going to write a function that validates workflows, (makes >>>>>> sure >>>>>> they are DAG's) and I was wondering if we should have a check for if >>>>>> the graph is connected or not. >>>>>> >>>>>> -Mike >>>>>> >>>>>> -------------------------------------------------------------------------- >>>>>> ---- >>>>>> Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT >>>>>> is a gathering of tech-side developers & brand creativity >>>>>> professionals. >>>>>> Meet >>>>>> the minds behind Google Creative Lab, Visual Complexity, >>>>>> Processing, & >>>>>> iPhoneDevCamp asthey present alongside digital heavyweights like >>>>>> Barbarian >>>>>> Group, R/GA, & Big Spaceship. http://www.creativitycat.com >>>>>> _______________________________________________ >>>>>> Pipeline130-developers mailing list >>>>>> Pip...@li... >>>>>> https://lists.sourceforge.net/lists/listinfo/pipeline130-developers >>>>>> >>>>> >>>>> --------------------------------------------------------------------------- >>>>> --- >>>>> Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT >>>>> is a gathering of tech-side developers & brand creativity >>>>> professionals. Meet >>>>> the minds behind Google Creative Lab, Visual Complexity, >>>>> Processing, & >>>>> iPhoneDevCamp asthey present alongside digital heavyweights like >>>>> Barbarian >>>>> Group, R/GA, & Big Spaceship. http://www.creativitycat.com >>>>> _______________________________________________ >>>>> Pipeline130-developers mailing list >>>>> Pip...@li... >>>>> https://lists.sourceforge.net/lists/listinfo/pipeline130-developers >>>> >>>> >>>> ---------------------------------------------------------------------------- >>>> -- >>>> Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT >>>> is a gathering of tech-side developers & brand creativity >>>> professionals. Meet >>>> the minds behind Google Creative Lab, Visual Complexity, Processing, & >>>> iPhoneDevCamp asthey present alongside digital heavyweights like >>>> Barbarian >>>> Group, R/GA, & Big Spaceship. http://www.creativitycat.com >>>> _______________________________________________ >>>> Pipeline130-developers mailing list >>>> Pip...@li... >>>> https://lists.sourceforge.net/lists/listinfo/pipeline130-developers >>> >>> >>> > ----------------------------------------------------------------------------->> > - >>> Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT >>> is a gathering of tech-side developers & brand creativity professionals. >>> Meet >>> the minds behind Google Creative Lab, Visual Complexity, Processing, & >>> iPhoneDevCamp asthey present alongside digital heavyweights like Barbarian >>> Group, R/GA, & Big Spaceship. http://www.creativitycat.com >>> _______________________________________________ >>> Pipeline130-developers mailing list >>> Pip...@li... >>> https://lists.sourceforge.net/lists/listinfo/pipeline130-developers >>> >> >> ------------------------------------------------------------------------------ >> Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT >> is a gathering of tech-side developers & brand creativity >> professionals. Meet >> the minds behind Google Creative Lab, Visual Complexity, Processing, & >> iPhoneDevCamp asthey present alongside digital heavyweights like Barbarian >> Group, R/GA, & Big Spaceship. http://www.creativitycat.com >> _______________________________________________ >> Pipeline130-developers mailing list >> Pip...@li... >> https://lists.sourceforge.net/lists/listinfo/pipeline130-developers > > > > ------------------------------------------------------------------------------ > Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT > is a gathering of tech-side developers & brand creativity professionals. Meet > the minds behind Google Creative Lab, Visual Complexity, Processing, & > iPhoneDevCamp asthey present alongside digital heavyweights like Barbarian > Group, R/GA, & Big Spaceship. http://www.creativitycat.com > _______________________________________________ > Pipeline130-developers mailing list > Pip...@li... > https://lists.sourceforge.net/lists/listinfo/pipeline130-developers > |
|
From: Mike H. <mh...@uc...> - 2009-05-21 06:41:09
|
I commited it to let you look at it but haven't tested it yet... Let me know what you think. I don't know what a depth first search is exactly but I'm almost 100% sure mine works in theory now. I think mine is worst case an O(n^2) if all jobs are disconnected from all other jobs, but in most cases should be O(n). I'm not at my computer anymore so those are off memory. I can probably come up with some test cases when I get back. How should these be made? Is it just a class that has a workflow that I know the problems with and just have a bunch of asserts? And for the status assertions should I just check that the components are correct or try to figure out a way to make sure it printed correctly also? Mike Hess On May 20, 2009, at 11:19 PM, "Chris Swan" <cs...@uc...> wrote: > If that doesn't work, look into using just a depth-first search... > pretty > sure you can make it O(n) if you do it right and shouldn't be too > hard to > implement. > > Someone: come up with good test cases for invalid workflows! > > -Chris > > -------------------------------------------------- > From: "Mike Hess" <mh...@uc...> > Sent: Wednesday, May 20, 2009 11:05 PM > To: <pip...@li...> > Subject: Re: [Pipeline130-developers] Validating Workflows > >> So basically the idea is the number of vertices is the number of jobs >> in the workflow, the number of edges is the sum of the number of >> children each job has, then the number of components is the hard >> part, >> but I added a pseudo-recursive function to check if is a new >> component. I am taking a math graph class currently alongside the cs >> class and I got the formula from the class (just took a test on it >> haha.) I'm not 100% sure the formula works both ways but I'll look >> into it deeper. >> >> >> -Mike >> >> On May 20, 2009, at 10:54 PM, Mike Hess wrote: >> >>> vertices = edges + components for an acyclic graph I'm pretty sure. >>> Should that work ok? Cuz I'm basically done with it right now, just >>> need to add comments and test it a little. >>> >>> -Mike >>> >>> On May 20, 2009, at 10:42 PM, Chris Swan wrote: >>> >>>> As long as the graph is acyclic, it's OK -- the unconnected parts >>>> will just >>>> run in parallel. What algorithm are you planning on using to detect >>>> cycles? >>>> >>>> -Chris >>>> >>>> -------------------------------------------------- >>>> From: "Mike Hess" <mh...@uc...> >>>> Sent: Wednesday, May 20, 2009 9:50 PM >>>> To: "CS 130" <pip...@li...> >>>> Subject: [Pipeline130-developers] Validating Workflows >>>> >>>>> I was going to write a function that validates workflows, (makes >>>>> sure >>>>> they are DAG's) and I was wondering if we should have a check >>>>> for if >>>>> the graph is connected or not. >>>>> >>>>> -Mike >>>>> >>>>> --- >>>>> --- >>>>> --- >>>>> --- >>>>> ------------------------------------------------------------------ >>>>> Register Now for Creativity and Technology (CaT), June 3rd, NYC. >>>>> CaT >>>>> is a gathering of tech-side developers & brand creativity >>>>> professionals. >>>>> Meet >>>>> the minds behind Google Creative Lab, Visual Complexity, >>>>> Processing, & >>>>> iPhoneDevCamp asthey present alongside digital heavyweights like >>>>> Barbarian >>>>> Group, R/GA, & Big Spaceship. http://www.creativitycat.com >>>>> _______________________________________________ >>>>> Pipeline130-developers mailing list >>>>> Pip...@li... >>>>> https://lists.sourceforge.net/lists/listinfo/pipeline130- >>>>> developers >>>>> >>>> >>>> --- >>>> --- >>>> --- >>>> --- >>>> ------------------------------------------------------------------ >>>> Register Now for Creativity and Technology (CaT), June 3rd, NYC. >>>> CaT >>>> is a gathering of tech-side developers & brand creativity >>>> professionals. Meet >>>> the minds behind Google Creative Lab, Visual Complexity, >>>> Processing, & >>>> iPhoneDevCamp asthey present alongside digital heavyweights like >>>> Barbarian >>>> Group, R/GA, & Big Spaceship. http://www.creativitycat.com >>>> _______________________________________________ >>>> Pipeline130-developers mailing list >>>> Pip...@li... >>>> https://lists.sourceforge.net/lists/listinfo/pipeline130-developers >>> >>> >>> --- >>> --- >>> --- >>> --- >>> ------------------------------------------------------------------ >>> Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT >>> is a gathering of tech-side developers & brand creativity >>> professionals. Meet >>> the minds behind Google Creative Lab, Visual Complexity, >>> Processing, & >>> iPhoneDevCamp asthey present alongside digital heavyweights like >>> Barbarian >>> Group, R/GA, & Big Spaceship. http://www.creativitycat.com >>> _______________________________________________ >>> Pipeline130-developers mailing list >>> Pip...@li... >>> https://lists.sourceforge.net/lists/listinfo/pipeline130-developers >> >> >> --- >> --- >> --- >> --------------------------------------------------------------------- >> Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT >> is a gathering of tech-side developers & brand creativity >> professionals. >> Meet >> the minds behind Google Creative Lab, Visual Complexity, >> Processing, & >> iPhoneDevCamp asthey present alongside digital heavyweights like >> Barbarian >> Group, R/GA, & Big Spaceship. http://www.creativitycat.com >> _______________________________________________ >> Pipeline130-developers mailing list >> Pip...@li... >> https://lists.sourceforge.net/lists/listinfo/pipeline130-developers >> > > --- > --- > --- > --------------------------------------------------------------------- > Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT > is a gathering of tech-side developers & brand creativity > professionals. Meet > the minds behind Google Creative Lab, Visual Complexity, Processing, & > iPhoneDevCamp asthey present alongside digital heavyweights like > Barbarian > Group, R/GA, & Big Spaceship. http://www.creativitycat.com > _______________________________________________ > Pipeline130-developers mailing list > Pip...@li... > https://lists.sourceforge.net/lists/listinfo/pipeline130-developers |
|
From: Nils H. <nil...@uc...> - 2009-05-21 06:35:53
|
Use DFS to find cycles; it's O(N) and it's easy to implement. Chris: what do you mean by invalid workflows? Do you mean cyclic graphs? Or do you mean workflows that fail (see the examples dir for some of these)? Nils On 5/20/09 11:19 PM, "Chris Swan" <cs...@uc...> wrote: > If that doesn't work, look into using just a depth-first search... pretty > sure you can make it O(n) if you do it right and shouldn't be too hard to > implement. > > Someone: come up with good test cases for invalid workflows! > > -Chris > > -------------------------------------------------- > From: "Mike Hess" <mh...@uc...> > Sent: Wednesday, May 20, 2009 11:05 PM > To: <pip...@li...> > Subject: Re: [Pipeline130-developers] Validating Workflows > >> So basically the idea is the number of vertices is the number of jobs >> in the workflow, the number of edges is the sum of the number of >> children each job has, then the number of components is the hard part, >> but I added a pseudo-recursive function to check if is a new >> component. I am taking a math graph class currently alongside the cs >> class and I got the formula from the class (just took a test on it >> haha.) I'm not 100% sure the formula works both ways but I'll look >> into it deeper. >> >> >> -Mike >> >> On May 20, 2009, at 10:54 PM, Mike Hess wrote: >> >>> vertices = edges + components for an acyclic graph I'm pretty sure. >>> Should that work ok? Cuz I'm basically done with it right now, just >>> need to add comments and test it a little. >>> >>> -Mike >>> >>> On May 20, 2009, at 10:42 PM, Chris Swan wrote: >>> >>>> As long as the graph is acyclic, it's OK -- the unconnected parts >>>> will just >>>> run in parallel. What algorithm are you planning on using to detect >>>> cycles? >>>> >>>> -Chris >>>> >>>> -------------------------------------------------- >>>> From: "Mike Hess" <mh...@uc...> >>>> Sent: Wednesday, May 20, 2009 9:50 PM >>>> To: "CS 130" <pip...@li...> >>>> Subject: [Pipeline130-developers] Validating Workflows >>>> >>>>> I was going to write a function that validates workflows, (makes >>>>> sure >>>>> they are DAG's) and I was wondering if we should have a check for if >>>>> the graph is connected or not. >>>>> >>>>> -Mike >>>>> >>>>> -------------------------------------------------------------------------- >>>>> ---- >>>>> Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT >>>>> is a gathering of tech-side developers & brand creativity >>>>> professionals. >>>>> Meet >>>>> the minds behind Google Creative Lab, Visual Complexity, >>>>> Processing, & >>>>> iPhoneDevCamp asthey present alongside digital heavyweights like >>>>> Barbarian >>>>> Group, R/GA, & Big Spaceship. http://www.creativitycat.com >>>>> _______________________________________________ >>>>> Pipeline130-developers mailing list >>>>> Pip...@li... >>>>> https://lists.sourceforge.net/lists/listinfo/pipeline130-developers >>>>> >>>> >>>> --------------------------------------------------------------------------- >>>> --- >>>> Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT >>>> is a gathering of tech-side developers & brand creativity >>>> professionals. Meet >>>> the minds behind Google Creative Lab, Visual Complexity, >>>> Processing, & >>>> iPhoneDevCamp asthey present alongside digital heavyweights like >>>> Barbarian >>>> Group, R/GA, & Big Spaceship. http://www.creativitycat.com >>>> _______________________________________________ >>>> Pipeline130-developers mailing list >>>> Pip...@li... >>>> https://lists.sourceforge.net/lists/listinfo/pipeline130-developers >>> >>> >>> ---------------------------------------------------------------------------- >>> -- >>> Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT >>> is a gathering of tech-side developers & brand creativity >>> professionals. Meet >>> the minds behind Google Creative Lab, Visual Complexity, Processing, & >>> iPhoneDevCamp asthey present alongside digital heavyweights like >>> Barbarian >>> Group, R/GA, & Big Spaceship. http://www.creativitycat.com >>> _______________________________________________ >>> Pipeline130-developers mailing list >>> Pip...@li... >>> https://lists.sourceforge.net/lists/listinfo/pipeline130-developers >> >> >> ----------------------------------------------------------------------------->> - >> Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT >> is a gathering of tech-side developers & brand creativity professionals. >> Meet >> the minds behind Google Creative Lab, Visual Complexity, Processing, & >> iPhoneDevCamp asthey present alongside digital heavyweights like Barbarian >> Group, R/GA, & Big Spaceship. http://www.creativitycat.com >> _______________________________________________ >> Pipeline130-developers mailing list >> Pip...@li... >> https://lists.sourceforge.net/lists/listinfo/pipeline130-developers >> > > ------------------------------------------------------------------------------ > Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT > is a gathering of tech-side developers & brand creativity professionals. Meet > the minds behind Google Creative Lab, Visual Complexity, Processing, & > iPhoneDevCamp asthey present alongside digital heavyweights like Barbarian > Group, R/GA, & Big Spaceship. http://www.creativitycat.com > _______________________________________________ > Pipeline130-developers mailing list > Pip...@li... > https://lists.sourceforge.net/lists/listinfo/pipeline130-developers |
|
From: Chris S. <cs...@uc...> - 2009-05-21 06:20:46
|
If that doesn't work, look into using just a depth-first search... pretty sure you can make it O(n) if you do it right and shouldn't be too hard to implement. Someone: come up with good test cases for invalid workflows! -Chris -------------------------------------------------- From: "Mike Hess" <mh...@uc...> Sent: Wednesday, May 20, 2009 11:05 PM To: <pip...@li...> Subject: Re: [Pipeline130-developers] Validating Workflows > So basically the idea is the number of vertices is the number of jobs > in the workflow, the number of edges is the sum of the number of > children each job has, then the number of components is the hard part, > but I added a pseudo-recursive function to check if is a new > component. I am taking a math graph class currently alongside the cs > class and I got the formula from the class (just took a test on it > haha.) I'm not 100% sure the formula works both ways but I'll look > into it deeper. > > > -Mike > > On May 20, 2009, at 10:54 PM, Mike Hess wrote: > >> vertices = edges + components for an acyclic graph I'm pretty sure. >> Should that work ok? Cuz I'm basically done with it right now, just >> need to add comments and test it a little. >> >> -Mike >> >> On May 20, 2009, at 10:42 PM, Chris Swan wrote: >> >>> As long as the graph is acyclic, it's OK -- the unconnected parts >>> will just >>> run in parallel. What algorithm are you planning on using to detect >>> cycles? >>> >>> -Chris >>> >>> -------------------------------------------------- >>> From: "Mike Hess" <mh...@uc...> >>> Sent: Wednesday, May 20, 2009 9:50 PM >>> To: "CS 130" <pip...@li...> >>> Subject: [Pipeline130-developers] Validating Workflows >>> >>>> I was going to write a function that validates workflows, (makes >>>> sure >>>> they are DAG's) and I was wondering if we should have a check for if >>>> the graph is connected or not. >>>> >>>> -Mike >>>> >>>> ------------------------------------------------------------------------------ >>>> Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT >>>> is a gathering of tech-side developers & brand creativity >>>> professionals. >>>> Meet >>>> the minds behind Google Creative Lab, Visual Complexity, >>>> Processing, & >>>> iPhoneDevCamp asthey present alongside digital heavyweights like >>>> Barbarian >>>> Group, R/GA, & Big Spaceship. http://www.creativitycat.com >>>> _______________________________________________ >>>> Pipeline130-developers mailing list >>>> Pip...@li... >>>> https://lists.sourceforge.net/lists/listinfo/pipeline130-developers >>>> >>> >>> ------------------------------------------------------------------------------ >>> Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT >>> is a gathering of tech-side developers & brand creativity >>> professionals. Meet >>> the minds behind Google Creative Lab, Visual Complexity, >>> Processing, & >>> iPhoneDevCamp asthey present alongside digital heavyweights like >>> Barbarian >>> Group, R/GA, & Big Spaceship. http://www.creativitycat.com >>> _______________________________________________ >>> Pipeline130-developers mailing list >>> Pip...@li... >>> https://lists.sourceforge.net/lists/listinfo/pipeline130-developers >> >> >> ------------------------------------------------------------------------------ >> Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT >> is a gathering of tech-side developers & brand creativity >> professionals. Meet >> the minds behind Google Creative Lab, Visual Complexity, Processing, & >> iPhoneDevCamp asthey present alongside digital heavyweights like >> Barbarian >> Group, R/GA, & Big Spaceship. http://www.creativitycat.com >> _______________________________________________ >> Pipeline130-developers mailing list >> Pip...@li... >> https://lists.sourceforge.net/lists/listinfo/pipeline130-developers > > > ------------------------------------------------------------------------------ > Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT > is a gathering of tech-side developers & brand creativity professionals. > Meet > the minds behind Google Creative Lab, Visual Complexity, Processing, & > iPhoneDevCamp asthey present alongside digital heavyweights like Barbarian > Group, R/GA, & Big Spaceship. http://www.creativitycat.com > _______________________________________________ > Pipeline130-developers mailing list > Pip...@li... > https://lists.sourceforge.net/lists/listinfo/pipeline130-developers > |
|
From: Mike H. <mh...@uc...> - 2009-05-21 06:06:25
|
So basically the idea is the number of vertices is the number of jobs in the workflow, the number of edges is the sum of the number of children each job has, then the number of components is the hard part, but I added a pseudo-recursive function to check if is a new component. I am taking a math graph class currently alongside the cs class and I got the formula from the class (just took a test on it haha.) I'm not 100% sure the formula works both ways but I'll look into it deeper. -Mike On May 20, 2009, at 10:54 PM, Mike Hess wrote: > vertices = edges + components for an acyclic graph I'm pretty sure. > Should that work ok? Cuz I'm basically done with it right now, just > need to add comments and test it a little. > > -Mike > > On May 20, 2009, at 10:42 PM, Chris Swan wrote: > >> As long as the graph is acyclic, it's OK -- the unconnected parts >> will just >> run in parallel. What algorithm are you planning on using to detect >> cycles? >> >> -Chris >> >> -------------------------------------------------- >> From: "Mike Hess" <mh...@uc...> >> Sent: Wednesday, May 20, 2009 9:50 PM >> To: "CS 130" <pip...@li...> >> Subject: [Pipeline130-developers] Validating Workflows >> >>> I was going to write a function that validates workflows, (makes >>> sure >>> they are DAG's) and I was wondering if we should have a check for if >>> the graph is connected or not. >>> >>> -Mike >>> >>> ------------------------------------------------------------------------------ >>> Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT >>> is a gathering of tech-side developers & brand creativity >>> professionals. >>> Meet >>> the minds behind Google Creative Lab, Visual Complexity, >>> Processing, & >>> iPhoneDevCamp asthey present alongside digital heavyweights like >>> Barbarian >>> Group, R/GA, & Big Spaceship. http://www.creativitycat.com >>> _______________________________________________ >>> Pipeline130-developers mailing list >>> Pip...@li... >>> https://lists.sourceforge.net/lists/listinfo/pipeline130-developers >>> >> >> ------------------------------------------------------------------------------ >> Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT >> is a gathering of tech-side developers & brand creativity >> professionals. Meet >> the minds behind Google Creative Lab, Visual Complexity, >> Processing, & >> iPhoneDevCamp asthey present alongside digital heavyweights like >> Barbarian >> Group, R/GA, & Big Spaceship. http://www.creativitycat.com >> _______________________________________________ >> Pipeline130-developers mailing list >> Pip...@li... >> https://lists.sourceforge.net/lists/listinfo/pipeline130-developers > > > ------------------------------------------------------------------------------ > Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT > is a gathering of tech-side developers & brand creativity > professionals. Meet > the minds behind Google Creative Lab, Visual Complexity, Processing, & > iPhoneDevCamp asthey present alongside digital heavyweights like > Barbarian > Group, R/GA, & Big Spaceship. http://www.creativitycat.com > _______________________________________________ > Pipeline130-developers mailing list > Pip...@li... > https://lists.sourceforge.net/lists/listinfo/pipeline130-developers |
|
From: Mike H. <mh...@uc...> - 2009-05-21 05:55:05
|
vertices = edges + components for an acyclic graph I'm pretty sure. Should that work ok? Cuz I'm basically done with it right now, just need to add comments and test it a little. -Mike On May 20, 2009, at 10:42 PM, Chris Swan wrote: > As long as the graph is acyclic, it's OK -- the unconnected parts > will just > run in parallel. What algorithm are you planning on using to detect > cycles? > > -Chris > > -------------------------------------------------- > From: "Mike Hess" <mh...@uc...> > Sent: Wednesday, May 20, 2009 9:50 PM > To: "CS 130" <pip...@li...> > Subject: [Pipeline130-developers] Validating Workflows > >> I was going to write a function that validates workflows, (makes sure >> they are DAG's) and I was wondering if we should have a check for if >> the graph is connected or not. >> >> -Mike >> >> ------------------------------------------------------------------------------ >> Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT >> is a gathering of tech-side developers & brand creativity >> professionals. >> Meet >> the minds behind Google Creative Lab, Visual Complexity, >> Processing, & >> iPhoneDevCamp asthey present alongside digital heavyweights like >> Barbarian >> Group, R/GA, & Big Spaceship. http://www.creativitycat.com >> _______________________________________________ >> Pipeline130-developers mailing list >> Pip...@li... >> https://lists.sourceforge.net/lists/listinfo/pipeline130-developers >> > > ------------------------------------------------------------------------------ > Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT > is a gathering of tech-side developers & brand creativity > professionals. Meet > the minds behind Google Creative Lab, Visual Complexity, Processing, & > iPhoneDevCamp asthey present alongside digital heavyweights like > Barbarian > Group, R/GA, & Big Spaceship. http://www.creativitycat.com > _______________________________________________ > Pipeline130-developers mailing list > Pip...@li... > https://lists.sourceforge.net/lists/listinfo/pipeline130-developers |
|
From: Chris S. <cs...@uc...> - 2009-05-21 05:42:35
|
As long as the graph is acyclic, it's OK -- the unconnected parts will just run in parallel. What algorithm are you planning on using to detect cycles? -Chris -------------------------------------------------- From: "Mike Hess" <mh...@uc...> Sent: Wednesday, May 20, 2009 9:50 PM To: "CS 130" <pip...@li...> Subject: [Pipeline130-developers] Validating Workflows > I was going to write a function that validates workflows, (makes sure > they are DAG's) and I was wondering if we should have a check for if > the graph is connected or not. > > -Mike > > ------------------------------------------------------------------------------ > Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT > is a gathering of tech-side developers & brand creativity professionals. > Meet > the minds behind Google Creative Lab, Visual Complexity, Processing, & > iPhoneDevCamp asthey present alongside digital heavyweights like Barbarian > Group, R/GA, & Big Spaceship. http://www.creativitycat.com > _______________________________________________ > Pipeline130-developers mailing list > Pip...@li... > https://lists.sourceforge.net/lists/listinfo/pipeline130-developers > |
|
From: Mike H. <mh...@uc...> - 2009-05-21 04:50:44
|
I was going to write a function that validates workflows, (makes sure they are DAG's) and I was wondering if we should have a check for if the graph is connected or not. -Mike |
|
From: Chris S. <cs...@uc...> - 2009-05-18 07:50:51
|
So it looks like things are allllllllmost working in my branch (branches/DRMAAConnector-ClusterInterface).
Here's what happens right now:
1) User starts engine on cluster computer with:
java -jar wfengine.jar
2) User submits a workflow using the manager with:
java -jar wfmanager.jar submit -e localhost -p 1300 -w workflow.xml
3) Every 20 seconds, the Engine starts any new workflows it has received and checks the status of all submitted jobs
4) Whenever a job is found to have completed, its children are notified and are free to begin
What's broken:
Looks like jobs work in reverse order. I don't know if this is an issue with the XML parsing or with the workflow/job classes.
I still have a bunch of work to do for tomorrow so I am just committing this. Please review my changes, because I made quite a few. I also apologize for not breaking it down into smaller commits, and for making a few unrelated changes.
ALSO: Compilation on the server seems to not work properly. I fixed that JUnit bug, but the Ant script seems to not put the proper Manifest file in the JAR it generates, and consequently, it can't find the WFLib library when you try to run it. The solution is to compile with Netbeans locally and transfer over the "dist" directory manually. You can find a working copy of this build in ~/cswan/built/dist on the server.
-Chris |
|
From: Mike H. <mh...@uc...> - 2009-05-17 05:05:18
|
good point. I doubt it really matters then. It doesn't seem like it will be that often that large quantities of jobs get started at the same time unless I am mistaken, but both seem to be about the same difficulty of implementation. I'd say just go with whatever. Mike Hess On May 16, 2009, at 9:49 PM, "Chris Swan" <cs...@uc...> wrote: > Not necessarily. We could easily implement a Job.GetReadyChildren() > that returns a list of child jobs that are ready to go thanks to its > parents completion, and execute all those jobs the same way. > > -Chris > > From: Mike Hess > Sent: Saturday, May 16, 2009 9:41 PM > To: pip...@li... > Subject: Re: [Pipeline130-developers] Workflows/jobs and > communication withcluster > > I would say option 1 because while it's efficient for the first run, > when a job completes and one new job needs to be run it seems like > in option two you would be iterating over all jobs in all workflows. > > Mike Hess > > On May 16, 2009, at 9:28 PM, "Chris Swan" <cs...@uc...> wrote: > >> Hey guys, >> >> I've begun attempting to connect the engine up with the cluster, >> but I've encountered a design problem and I'd really like to hear >> your feedback on it. >> >> Right now, we use a model where the Engine tells all of its >> Workflows, "start your ready jobs". The workflows then tell each of >> its jobs, "start if you are ready". Right now we've implemented >> that just fine except for the actual transmission of the job. In >> order for each Job to autonomously submit itself to the cluster, >> each Job must know about the DRMAAConnector (my name for the >> cluster interface). However, Jobs and Workflows are part of WFLib, >> shared by the Manager, Engine, and Editor. >> >> I see two immediate solutions for this: >> 1) We define a new interface in WFLib, ClusterInterface, which >> contains the method "SubmitJob(Job j)". Workflow.StartReadyJobs and >> Job.StartIfReady() will be modified to take a ClusterInterface and >> invoke its SubmitJob method on themselves. DRMAAConnector will >> implement this interface. >> >> 2) We change the "Start" methods to be "Get" methods that return a >> list of ready jobs. This way, we can perform a query over all >> workflows, generate a list of all the ready-to-run jobs, and submit >> all the jobs in one shot using DRMAA's batch submit functionality. >> This will greatly reduce overhead, but it's less pleasant from >> an abstraction level. >> >> What do you guys think? I think I will be making a branch in order >> to prototype one of the above solutions. Please respond to this >> email with an opinion. If you do not understand something, please >> say so so we can get you up to speed. It is important that we all >> understand the project and its design decisions. >> >> -Chris > >> --- >> --- >> --- >> --------------------------------------------------------------------- >> Crystal Reports - New Free Runtime and 30 Day Trial >> Check out the new simplified licensing option that enables >> unlimited royalty-free distribution of the report engine >> for externally facing server and web deployment. >> http://p.sf.net/sfu/businessobjects > >> _______________________________________________ >> Pipeline130-developers mailing list >> Pip...@li... >> https://lists.sourceforge.net/lists/listinfo/pipeline130-developers > > --- > --- > --- > --------------------------------------------------------------------- > Crystal Reports - New Free Runtime and 30 Day Trial > Check out the new simplified licensing option that enables > unlimited royalty-free distribution of the report engine > for externally facing server and web deployment. > http://p.sf.net/sfu/businessobjects > _______________________________________________ > Pipeline130-developers mailing list > Pip...@li... > https://lists.sourceforge.net/lists/listinfo/pipeline130-developers > --- > --- > --- > --------------------------------------------------------------------- > Crystal Reports - New Free Runtime and 30 Day Trial > Check out the new simplified licensing option that enables > unlimited royalty-free distribution of the report engine > for externally facing server and web deployment. > http://p.sf.net/sfu/businessobjects > _______________________________________________ > Pipeline130-developers mailing list > Pip...@li... > https://lists.sourceforge.net/lists/listinfo/pipeline130-developers |
|
From: Nils H. <nil...@uc...> - 2009-05-17 05:02:15
|
I would say option 1 since then the jobs and workflows still don¹t interact directly with DRMAA. On 5/16/09 9:28 PM, "Chris Swan" <cs...@uc...> wrote: > Hey guys, > > I've begun attempting to connect the engine up with the cluster, but I've > encountered a design problem and I'd really like to hear your feedback on it. > > Right now, we use a model where the Engine tells all of its Workflows, "start > your ready jobs". The workflows then tell each of its jobs, "start if you are > ready". Right now we've implemented that just fine except for the actual > transmission of the job. In order for each Job to autonomously submit itself > to the cluster, each Job must know about the DRMAAConnector (my name for the > cluster interface). However, Jobs and Workflows are part of WFLib, shared by > the Manager, Engine, and Editor. > > I see two immediate solutions for this: > 1) We define a new interface in WFLib, ClusterInterface, which contains the > method "SubmitJob(Job j)". Workflow.StartReadyJobs and Job.StartIfReady() will > be modified to take a ClusterInterface and invoke its SubmitJob method on > themselves. DRMAAConnector will implement this interface. > > 2) We change the "Start" methods to be "Get" methods that return a list of > ready jobs. This way, we can perform a query over all workflows, generate a > list of all the ready-to-run jobs, and submit all the jobs in one shot using > DRMAA's batch submit functionality. This will greatly reduce overhead, but > it's less pleasant from an abstraction level. > > What do you guys think? I think I will be making a branch in order to > prototype one of the above solutions. Please respond to this email with an > opinion. If you do not understand something, please say so so we can get you > up to speed. It is important that we all understand the project and its design > decisions. > > -Chris > > > ------------------------------------------------------------------------------ > Crystal Reports - New Free Runtime and 30 Day Trial > Check out the new simplified licensing option that enables > unlimited royalty-free distribution of the report engine > for externally facing server and web deployment. > http://p.sf.net/sfu/businessobjects > > _______________________________________________ > Pipeline130-developers mailing list > Pip...@li... > https://lists.sourceforge.net/lists/listinfo/pipeline130-developers |
|
From: Chris S. <cs...@uc...> - 2009-05-17 04:50:12
|
Not necessarily. We could easily implement a Job.GetReadyChildren() that returns a list of child jobs that are ready to go thanks to its parents completion, and execute all those jobs the same way. -Chris From: Mike Hess Sent: Saturday, May 16, 2009 9:41 PM To: pip...@li... Subject: Re: [Pipeline130-developers] Workflows/jobs and communication withcluster I would say option 1 because while it's efficient for the first run, when a job completes and one new job needs to be run it seems like in option two you would be iterating over all jobs in all workflows. Mike Hess On May 16, 2009, at 9:28 PM, "Chris Swan" <cs...@uc...> wrote: Hey guys, I've begun attempting to connect the engine up with the cluster, but I've encountered a design problem and I'd really like to hear your feedback on it. Right now, we use a model where the Engine tells all of its Workflows, "start your ready jobs". The workflows then tell each of its jobs, "start if you are ready". Right now we've implemented that just fine except for the actual transmission of the job. In order for each Job to autonomously submit itself to the cluster, each Job must know about the DRMAAConnector (my name for the cluster interface). However, Jobs and Workflows are part of WFLib, shared by the Manager, Engine, and Editor. I see two immediate solutions for this: 1) We define a new interface in WFLib, ClusterInterface, which contains the method "SubmitJob(Job j)". Workflow.StartReadyJobs and Job.StartIfReady() will be modified to take a ClusterInterface and invoke its SubmitJob method on themselves. DRMAAConnector will implement this interface. 2) We change the "Start" methods to be "Get" methods that return a list of ready jobs. This way, we can perform a query over all workflows, generate a list of all the ready-to-run jobs, and submit all the jobs in one shot using DRMAA's batch submit functionality. This will greatly reduce overhead, but it's less pleasant from an abstraction level. What do you guys think? I think I will be making a branch in order to prototype one of the above solutions. Please respond to this email with an opinion. If you do not understand something, please say so so we can get you up to speed. It is important that we all understand the project and its design decisions. -Chris ------------------------------------------------------------------------------ Crystal Reports - New Free Runtime and 30 Day Trial Check out the new simplified licensing option that enables unlimited royalty-free distribution of the report engine for externally facing server and web deployment. http://p.sf.net/sfu/businessobjects _______________________________________________ Pipeline130-developers mailing list Pip...@li... https://lists.sourceforge.net/lists/listinfo/pipeline130-developers -------------------------------------------------------------------------------- ------------------------------------------------------------------------------ Crystal Reports - New Free Runtime and 30 Day Trial Check out the new simplified licensing option that enables unlimited royalty-free distribution of the report engine for externally facing server and web deployment. http://p.sf.net/sfu/businessobjects -------------------------------------------------------------------------------- _______________________________________________ Pipeline130-developers mailing list Pip...@li... https://lists.sourceforge.net/lists/listinfo/pipeline130-developers |
|
From: Mike H. <mh...@uc...> - 2009-05-17 04:41:59
|
I would say option 1 because while it's efficient for the first run, when a job completes and one new job needs to be run it seems like in option two you would be iterating over all jobs in all workflows. Mike Hess On May 16, 2009, at 9:28 PM, "Chris Swan" <cs...@uc...> wrote: > Hey guys, > > I've begun attempting to connect the engine up with the cluster, but > I've encountered a design problem and I'd really like to hear your > feedback on it. > > Right now, we use a model where the Engine tells all of its > Workflows, "start your ready jobs". The workflows then tell each of > its jobs, "start if you are ready". Right now we've implemented that > just fine except for the actual transmission of the job. In order > for each Job to autonomously submit itself to the cluster, each Job > must know about the DRMAAConnector (my name for the cluster > interface). However, Jobs and Workflows are part of WFLib, shared by > the Manager, Engine, and Editor. > > I see two immediate solutions for this: > 1) We define a new interface in WFLib, ClusterInterface, which > contains the method "SubmitJob(Job j)". Workflow.StartReadyJobs and > Job.StartIfReady() will be modified to take a ClusterInterface and > invoke its SubmitJob method on themselves. DRMAAConnector will > implement this interface. > > 2) We change the "Start" methods to be "Get" methods that return a > list of ready jobs. This way, we can perform a query over all > workflows, generate a list of all the ready-to-run jobs, and submit > all the jobs in one shot using DRMAA's batch submit functionality. > This will greatly reduce overhead, but it's less pleasant from an > abstraction level. > > What do you guys think? I think I will be making a branch in order > to prototype one of the above solutions. Please respond to this > email with an opinion. If you do not understand something, please > say so so we can get you up to speed. It is important that we all > understand the project and its design decisions. > > -Chris > --- > --- > --- > --------------------------------------------------------------------- > Crystal Reports - New Free Runtime and 30 Day Trial > Check out the new simplified licensing option that enables > unlimited royalty-free distribution of the report engine > for externally facing server and web deployment. > http://p.sf.net/sfu/businessobjects > _______________________________________________ > Pipeline130-developers mailing list > Pip...@li... > https://lists.sourceforge.net/lists/listinfo/pipeline130-developers |
|
From: Chris S. <cs...@uc...> - 2009-05-17 04:28:53
|
Hey guys, I've begun attempting to connect the engine up with the cluster, but I've encountered a design problem and I'd really like to hear your feedback on it. Right now, we use a model where the Engine tells all of its Workflows, "start your ready jobs". The workflows then tell each of its jobs, "start if you are ready". Right now we've implemented that just fine except for the actual transmission of the job. In order for each Job to autonomously submit itself to the cluster, each Job must know about the DRMAAConnector (my name for the cluster interface). However, Jobs and Workflows are part of WFLib, shared by the Manager, Engine, and Editor. I see two immediate solutions for this: 1) We define a new interface in WFLib, ClusterInterface, which contains the method "SubmitJob(Job j)". Workflow.StartReadyJobs and Job.StartIfReady() will be modified to take a ClusterInterface and invoke its SubmitJob method on themselves. DRMAAConnector will implement this interface. 2) We change the "Start" methods to be "Get" methods that return a list of ready jobs. This way, we can perform a query over all workflows, generate a list of all the ready-to-run jobs, and submit all the jobs in one shot using DRMAA's batch submit functionality. This will greatly reduce overhead, but it's less pleasant from an abstraction level. What do you guys think? I think I will be making a branch in order to prototype one of the above solutions. Please respond to this email with an opinion. If you do not understand something, please say so so we can get you up to speed. It is important that we all understand the project and its design decisions. -Chris |
|
From: Chris S. <cs...@uc...> - 2009-05-16 20:45:12
|
Figured it out -- you can only see the status of finished jobs from within the session you started it in. So we will definitely have to persist our session information through "Session.getContact()" and "Session.init(contact)" even though tools like qstat, etc can see across session borders. -Chris -------------------------------------------------- From: "Chris Swan" <cs...@uc...> Sent: Friday, May 15, 2009 11:28 AM To: <pip...@li...> Subject: Re: [Pipeline130-developers] for generating unique ids > That's what I thought, but I wrote up a test case that submits a job and > does not wait for it, and a separate program that just calls > getJobProgramStatus. Once the job finishes, getJobProgramStatus starts > failing, even though nobody had waited on the job. > > According to the docs it seems like that should work, so maybe I was doing > something wrong... I guess I'll investigate more. > > -Chris > > -------------------------------------------------- > From: "Nils Homer" <nil...@uc...> > Sent: Friday, May 15, 2009 9:05 AM > To: <pip...@li...> > Subject: Re: [Pipeline130-developers] for generating unique ids > >> Here's the solution: a job is never "disposed" until "wait()" routine is >> called. Therefore when you execute a "wait" command, you should locally >> store the result. Any other query on that job should check the local >> variable not query the cluster. >> >> For example, you could use "getJobProgramStatus" to see if a "wait()" is >> ready to be called. I think having each "wait" as a new thread is not a >> good idea, when there could be thousands of jobs (thus thousands of waits >> and threads). >> >> Nils >> >> >> On 5/14/09 11:29 PM, "Chris Swan" <cs...@uc...> wrote: >> >>> Re-sending this message to the list instead of straight to Nils: >>> >>> >>> Do you know how to do this through DRMAA, though? >>> >>> There seems to be a concept of "reaped" jobs. After a job is waited >>> on, it is considered "reaped" and cannot be accessed any more. >>> However, in my experiments so far (starting a job and then not waiting >>> on it), the job disappears even if we don't call wait(). >>> Session.getJobProgramStatus() returns values while the job is queued >>> and while it is running, but once the job terminates, it just says >>> that the job does not exist. >>> >>> Whether or not we can get information from jobs after they have >>> terminated will shape the design of the entire Engine, so we need to >>> figure this out soon. >>> >>> -Chris >>> >>> Quoting Nils Homer <nil...@uc...>: >>> >>>> 2. If you ever want to manually see the status of a completed job, use >>>> "qacct -j $JOB_ID". Thus, the job never disappears! >>>> >>>> Nils >>> >>> >>> ------------------------------------------------------------------------------ >>> Crystal Reports - New Free Runtime and 30 Day Trial >>> Check out the new simplified licensing option that enables >>> unlimited royalty-free distribution of the report engine >>> for externally facing server and web deployment. >>> http://p.sf.net/sfu/businessobjects >>> _______________________________________________ >>> Pipeline130-developers mailing list >>> Pip...@li... >>> https://lists.sourceforge.net/lists/listinfo/pipeline130-developers >> >> >> >> ------------------------------------------------------------------------------ >> Crystal Reports - New Free Runtime and 30 Day Trial >> Check out the new simplified licensing option that enables >> unlimited royalty-free distribution of the report engine >> for externally facing server and web deployment. >> http://p.sf.net/sfu/businessobjects >> _______________________________________________ >> Pipeline130-developers mailing list >> Pip...@li... >> https://lists.sourceforge.net/lists/listinfo/pipeline130-developers >> > > ------------------------------------------------------------------------------ > Crystal Reports - New Free Runtime and 30 Day Trial > Check out the new simplified licensing option that enables > unlimited royalty-free distribution of the report engine > for externally facing server and web deployment. > http://p.sf.net/sfu/businessobjects > _______________________________________________ > Pipeline130-developers mailing list > Pip...@li... > https://lists.sourceforge.net/lists/listinfo/pipeline130-developers > |
|
From: Chris S. <cs...@uc...> - 2009-05-15 18:28:41
|
That's what I thought, but I wrote up a test case that submits a job and does not wait for it, and a separate program that just calls getJobProgramStatus. Once the job finishes, getJobProgramStatus starts failing, even though nobody had waited on the job. According to the docs it seems like that should work, so maybe I was doing something wrong... I guess I'll investigate more. -Chris -------------------------------------------------- From: "Nils Homer" <nil...@uc...> Sent: Friday, May 15, 2009 9:05 AM To: <pip...@li...> Subject: Re: [Pipeline130-developers] for generating unique ids > Here's the solution: a job is never "disposed" until "wait()" routine is > called. Therefore when you execute a "wait" command, you should locally > store the result. Any other query on that job should check the local > variable not query the cluster. > > For example, you could use "getJobProgramStatus" to see if a "wait()" is > ready to be called. I think having each "wait" as a new thread is not a > good idea, when there could be thousands of jobs (thus thousands of waits > and threads). > > Nils > > > On 5/14/09 11:29 PM, "Chris Swan" <cs...@uc...> wrote: > >> Re-sending this message to the list instead of straight to Nils: >> >> >> Do you know how to do this through DRMAA, though? >> >> There seems to be a concept of "reaped" jobs. After a job is waited >> on, it is considered "reaped" and cannot be accessed any more. >> However, in my experiments so far (starting a job and then not waiting >> on it), the job disappears even if we don't call wait(). >> Session.getJobProgramStatus() returns values while the job is queued >> and while it is running, but once the job terminates, it just says >> that the job does not exist. >> >> Whether or not we can get information from jobs after they have >> terminated will shape the design of the entire Engine, so we need to >> figure this out soon. >> >> -Chris >> >> Quoting Nils Homer <nil...@uc...>: >> >>> 2. If you ever want to manually see the status of a completed job, use >>> "qacct -j $JOB_ID". Thus, the job never disappears! >>> >>> Nils >> >> >> ------------------------------------------------------------------------------ >> Crystal Reports - New Free Runtime and 30 Day Trial >> Check out the new simplified licensing option that enables >> unlimited royalty-free distribution of the report engine >> for externally facing server and web deployment. >> http://p.sf.net/sfu/businessobjects >> _______________________________________________ >> Pipeline130-developers mailing list >> Pip...@li... >> https://lists.sourceforge.net/lists/listinfo/pipeline130-developers > > > > ------------------------------------------------------------------------------ > Crystal Reports - New Free Runtime and 30 Day Trial > Check out the new simplified licensing option that enables > unlimited royalty-free distribution of the report engine > for externally facing server and web deployment. > http://p.sf.net/sfu/businessobjects > _______________________________________________ > Pipeline130-developers mailing list > Pip...@li... > https://lists.sourceforge.net/lists/listinfo/pipeline130-developers > |
|
From: Nils H. <nil...@uc...> - 2009-05-15 16:06:38
|
Here's the solution: a job is never "disposed" until "wait()" routine is called. Therefore when you execute a "wait" command, you should locally store the result. Any other query on that job should check the local variable not query the cluster. For example, you could use "getJobProgramStatus" to see if a "wait()" is ready to be called. I think having each "wait" as a new thread is not a good idea, when there could be thousands of jobs (thus thousands of waits and threads). Nils On 5/14/09 11:29 PM, "Chris Swan" <cs...@uc...> wrote: > Re-sending this message to the list instead of straight to Nils: > > > Do you know how to do this through DRMAA, though? > > There seems to be a concept of "reaped" jobs. After a job is waited > on, it is considered "reaped" and cannot be accessed any more. > However, in my experiments so far (starting a job and then not waiting > on it), the job disappears even if we don't call wait(). > Session.getJobProgramStatus() returns values while the job is queued > and while it is running, but once the job terminates, it just says > that the job does not exist. > > Whether or not we can get information from jobs after they have > terminated will shape the design of the entire Engine, so we need to > figure this out soon. > > -Chris > > Quoting Nils Homer <nil...@uc...>: > >> 2. If you ever want to manually see the status of a completed job, use >> "qacct -j $JOB_ID". Thus, the job never disappears! >> >> Nils > > > ------------------------------------------------------------------------------ > Crystal Reports - New Free Runtime and 30 Day Trial > Check out the new simplified licensing option that enables > unlimited royalty-free distribution of the report engine > for externally facing server and web deployment. > http://p.sf.net/sfu/businessobjects > _______________________________________________ > Pipeline130-developers mailing list > Pip...@li... > https://lists.sourceforge.net/lists/listinfo/pipeline130-developers |
|
From: Chris S. <cs...@uc...> - 2009-05-15 06:30:26
|
Re-sending this message to the list instead of straight to Nils: Do you know how to do this through DRMAA, though? There seems to be a concept of "reaped" jobs. After a job is waited on, it is considered "reaped" and cannot be accessed any more. However, in my experiments so far (starting a job and then not waiting on it), the job disappears even if we don't call wait(). Session.getJobProgramStatus() returns values while the job is queued and while it is running, but once the job terminates, it just says that the job does not exist. Whether or not we can get information from jobs after they have terminated will shape the design of the entire Engine, so we need to figure this out soon. -Chris Quoting Nils Homer <nil...@uc...>: > 2. If you ever want to manually see the status of a completed job, use > "qacct -j $JOB_ID". Thus, the job never disappears! > > Nils |
|
From: Chris S. <cs...@uc...> - 2009-05-14 05:03:26
|
Ah, I only knew about qstat. Good to know. To generate unique IDs we should use a unique identifier generator library. Looks like the JDK already has one: http://java.sun.com/j2se/1.5.0/docs/api/java/util/UUID.html At some point we definitely will want to store the current state. During our meeting in Powell we discussed storing the base workflow data structure plus "journal" events for when a job gets an ID on the cluster. From the job #s and the overall workflow structure, we can reconstruct the latest state of the workflow by querying the cluster for each job's status. -Chris -------------------------------------------------- From: "Nils Homer" <nil...@uc...> Sent: Wednesday, May 13, 2009 8:43 PM To: <pip...@li...> Subject: Re: [Pipeline130-developers] for generating unique ids > 1. We should definitely store all current workflows as well as their qsub > IDs. > > 2. If you ever want to manually see the status of a completed job, use > "qacct -j $JOB_ID". Thus, the job never disappears! > > Nils > > > On 5/13/09 8:37 PM, "Mike Hess" <mh...@uc...> wrote: > >> I was wondering how you guys were thinking about doing that. >> Personally I was thinking maybe have a file stored somewhere that just >> keeps the last key generated, then just increment that key to get the >> next identifier. It should be fairly easy to implement and should >> ensure that no identifiers are used twice. Thoughts? >> >> I also think we are also going to need a folder that stores the >> current running workflows if I am not mistaken so that in the event >> that the engine crashes, it can be restarted in the state it was >> before. Is this how the system should work? >> >> >> -Mike >> >> ------------------------------------------------------------------------------ >> The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your >> production scanning environment may not be a perfect world - but thanks >> to >> Kodak, there's a perfect scanner to get the job done! With the NEW KODAK >> i700 >> Series Scanner you'll get full speed at 300 dpi even with all image >> processing features enabled. http://p.sf.net/sfu/kodak-com >> _______________________________________________ >> Pipeline130-developers mailing list >> Pip...@li... >> https://lists.sourceforge.net/lists/listinfo/pipeline130-developers > > > > ------------------------------------------------------------------------------ > The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your > production scanning environment may not be a perfect world - but thanks to > Kodak, there's a perfect scanner to get the job done! With the NEW KODAK > i700 > Series Scanner you'll get full speed at 300 dpi even with all image > processing features enabled. http://p.sf.net/sfu/kodak-com > _______________________________________________ > Pipeline130-developers mailing list > Pip...@li... > https://lists.sourceforge.net/lists/listinfo/pipeline130-developers > |
|
From: Nils H. <nil...@uc...> - 2009-05-14 03:43:53
|
1. We should definitely store all current workflows as well as their qsub IDs. 2. If you ever want to manually see the status of a completed job, use "qacct -j $JOB_ID". Thus, the job never disappears! Nils On 5/13/09 8:37 PM, "Mike Hess" <mh...@uc...> wrote: > I was wondering how you guys were thinking about doing that. > Personally I was thinking maybe have a file stored somewhere that just > keeps the last key generated, then just increment that key to get the > next identifier. It should be fairly easy to implement and should > ensure that no identifiers are used twice. Thoughts? > > I also think we are also going to need a folder that stores the > current running workflows if I am not mistaken so that in the event > that the engine crashes, it can be restarted in the state it was > before. Is this how the system should work? > > > -Mike > > ------------------------------------------------------------------------------ > The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your > production scanning environment may not be a perfect world - but thanks to > Kodak, there's a perfect scanner to get the job done! With the NEW KODAK i700 > Series Scanner you'll get full speed at 300 dpi even with all image > processing features enabled. http://p.sf.net/sfu/kodak-com > _______________________________________________ > Pipeline130-developers mailing list > Pip...@li... > https://lists.sourceforge.net/lists/listinfo/pipeline130-developers |
|
From: Mike H. <mh...@uc...> - 2009-05-14 03:37:29
|
I was wondering how you guys were thinking about doing that. Personally I was thinking maybe have a file stored somewhere that just keeps the last key generated, then just increment that key to get the next identifier. It should be fairly easy to implement and should ensure that no identifiers are used twice. Thoughts? I also think we are also going to need a folder that stores the current running workflows if I am not mistaken so that in the event that the engine crashes, it can be restarted in the state it was before. Is this how the system should work? -Mike |
|
From: Chris S. <cs...@uc...> - 2009-05-14 03:20:48
|
Awesome! After looking into this stuff a bit more, I think we might need to
slightly rethink the way the Engine works. I don't know if we can just poll
since jobs that have completed simply disappear from SGE altogether.
However, I think if we use the "synchronize" functionality, or set it to
wait for "all jobs" rather than just one, from a dedicated thread, that
might work.
-Chris
--------------------------------------------------
From: "Nils Homer" <nil...@uc...>
Sent: Wednesday, May 13, 2009 8:13 PM
To: <pip...@li...>
Subject: Re: [Pipeline130-developers] DRMAA
> Ok, I figured it out.
>
> We need to set the "Native Specification" to:
>
> jt.setNativeSpecification("-b n");
>
> Reference:
> http://markmail.org/message/x7cubdcw4nduv75y#query:+page:1+mid:xzavt7onfvdne
> rz2+state:results
>
> We can now begin to implement the DRMAA interface. I will start coding
> this
> tonight,
>
> Nils
>
>
> On 5/13/09 7:24 PM, "Chris Swan" <cs...@uc...> wrote:
>
>> I really am stumped by this. I can submit a job manually with:
>> qsub
>> /export/home/cs130/cswan/pipeline130/trunk/sandbox/drmaa/helloworld.sh
>> and check its status with
>> qstat -j <number>
>>
>> The job runs perfectly (I made it so it touches the file "testfile" in
>> the
>> source directory). Then, using DRMAA, I submit an identical job. Running
>> qstat -j on this job gives me identical information to the qsub version,
>> but
>> the script does not get executed.
>>
>> No idea what's going on. Probably won't be able to look at it any more
>> tonight though.
>>
>> -Chris
>>
>> --------------------------------------------------
>> From: "Nils Homer" <nil...@uc...>
>> Sent: Wednesday, May 13, 2009 5:56 PM
>> To: <pip...@li...>
>> Subject: [Pipeline130-developers] DRMAA
>>
>>> Just to let everyone know, I am still testing out the DRMAA interface
>>> via
>>> the drmaa example in the sandbox folder (note: there are hard-coded
>>> links).
>>> Anyhow, if someone figures out how to get a job to successfully run
>>> (mine
>>> are always aborted after submission), let me know.
>>>
>>> Nis
>>>
>>>
>>>
>>>
> ----------------------------------------------------------------------------->>
> -
>>> The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your
>>> production scanning environment may not be a perfect world - but thanks
>>> to
>>> Kodak, there's a perfect scanner to get the job done! With the NEW KODAK
>>> i700
>>> Series Scanner you'll get full speed at 300 dpi even with all image
>>> processing features enabled. http://p.sf.net/sfu/kodak-com
>>> _______________________________________________
>>> Pipeline130-developers mailing list
>>> Pip...@li...
>>> https://lists.sourceforge.net/lists/listinfo/pipeline130-developers
>>>
>>
>> ------------------------------------------------------------------------------
>> The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your
>> production scanning environment may not be a perfect world - but thanks
>> to
>> Kodak, there's a perfect scanner to get the job done! With the NEW KODAK
>> i700
>> Series Scanner you'll get full speed at 300 dpi even with all image
>> processing features enabled. http://p.sf.net/sfu/kodak-com
>> _______________________________________________
>> Pipeline130-developers mailing list
>> Pip...@li...
>> https://lists.sourceforge.net/lists/listinfo/pipeline130-developers
>
>
>
> ------------------------------------------------------------------------------
> The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your
> production scanning environment may not be a perfect world - but thanks to
> Kodak, there's a perfect scanner to get the job done! With the NEW KODAK
> i700
> Series Scanner you'll get full speed at 300 dpi even with all image
> processing features enabled. http://p.sf.net/sfu/kodak-com
> _______________________________________________
> Pipeline130-developers mailing list
> Pip...@li...
> https://lists.sourceforge.net/lists/listinfo/pipeline130-developers
>
|
|
From: Nils H. <nil...@uc...> - 2009-05-14 03:14:26
|
Ok, I figured it out.
We need to set the "Native Specification" to:
jt.setNativeSpecification("-b n");
Reference:
http://markmail.org/message/x7cubdcw4nduv75y#query:+page:1+mid:xzavt7onfvdne
rz2+state:results
We can now begin to implement the DRMAA interface. I will start coding this
tonight,
Nils
On 5/13/09 7:24 PM, "Chris Swan" <cs...@uc...> wrote:
> I really am stumped by this. I can submit a job manually with:
> qsub /export/home/cs130/cswan/pipeline130/trunk/sandbox/drmaa/helloworld.sh
> and check its status with
> qstat -j <number>
>
> The job runs perfectly (I made it so it touches the file "testfile" in the
> source directory). Then, using DRMAA, I submit an identical job. Running
> qstat -j on this job gives me identical information to the qsub version, but
> the script does not get executed.
>
> No idea what's going on. Probably won't be able to look at it any more
> tonight though.
>
> -Chris
>
> --------------------------------------------------
> From: "Nils Homer" <nil...@uc...>
> Sent: Wednesday, May 13, 2009 5:56 PM
> To: <pip...@li...>
> Subject: [Pipeline130-developers] DRMAA
>
>> Just to let everyone know, I am still testing out the DRMAA interface via
>> the drmaa example in the sandbox folder (note: there are hard-coded
>> links).
>> Anyhow, if someone figures out how to get a job to successfully run (mine
>> are always aborted after submission), let me know.
>>
>> Nis
>>
>>
>>
>>
----------------------------------------------------------------------------->>
-
>> The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your
>> production scanning environment may not be a perfect world - but thanks to
>> Kodak, there's a perfect scanner to get the job done! With the NEW KODAK
>> i700
>> Series Scanner you'll get full speed at 300 dpi even with all image
>> processing features enabled. http://p.sf.net/sfu/kodak-com
>> _______________________________________________
>> Pipeline130-developers mailing list
>> Pip...@li...
>> https://lists.sourceforge.net/lists/listinfo/pipeline130-developers
>>
>
> ------------------------------------------------------------------------------
> The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your
> production scanning environment may not be a perfect world - but thanks to
> Kodak, there's a perfect scanner to get the job done! With the NEW KODAK i700
> Series Scanner you'll get full speed at 300 dpi even with all image
> processing features enabled. http://p.sf.net/sfu/kodak-com
> _______________________________________________
> Pipeline130-developers mailing list
> Pip...@li...
> https://lists.sourceforge.net/lists/listinfo/pipeline130-developers
|
|
From: Nils H. <nil...@uc...> - 2009-05-14 02:59:21
|
There's something I'm missing too. I will figure it out tonight. On 5/13/09 7:24 PM, "Chris Swan" <cs...@uc...> wrote: > I really am stumped by this. I can submit a job manually with: > qsub /export/home/cs130/cswan/pipeline130/trunk/sandbox/drmaa/helloworld.sh > and check its status with > qstat -j <number> > > The job runs perfectly (I made it so it touches the file "testfile" in the > source directory). Then, using DRMAA, I submit an identical job. Running > qstat -j on this job gives me identical information to the qsub version, but > the script does not get executed. > > No idea what's going on. Probably won't be able to look at it any more > tonight though. > > -Chris > > -------------------------------------------------- > From: "Nils Homer" <nil...@uc...> > Sent: Wednesday, May 13, 2009 5:56 PM > To: <pip...@li...> > Subject: [Pipeline130-developers] DRMAA > >> Just to let everyone know, I am still testing out the DRMAA interface via >> the drmaa example in the sandbox folder (note: there are hard-coded >> links). >> Anyhow, if someone figures out how to get a job to successfully run (mine >> are always aborted after submission), let me know. >> >> Nis >> >> >> >> ----------------------------------------------------------------------------->> - >> The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your >> production scanning environment may not be a perfect world - but thanks to >> Kodak, there's a perfect scanner to get the job done! With the NEW KODAK >> i700 >> Series Scanner you'll get full speed at 300 dpi even with all image >> processing features enabled. http://p.sf.net/sfu/kodak-com >> _______________________________________________ >> Pipeline130-developers mailing list >> Pip...@li... >> https://lists.sourceforge.net/lists/listinfo/pipeline130-developers >> > > ------------------------------------------------------------------------------ > The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your > production scanning environment may not be a perfect world - but thanks to > Kodak, there's a perfect scanner to get the job done! With the NEW KODAK i700 > Series Scanner you'll get full speed at 300 dpi even with all image > processing features enabled. http://p.sf.net/sfu/kodak-com > _______________________________________________ > Pipeline130-developers mailing list > Pip...@li... > https://lists.sourceforge.net/lists/listinfo/pipeline130-developers |
|
From: Chris S. <cs...@uc...> - 2009-05-14 02:25:38
|
I really am stumped by this. I can submit a job manually with: qsub /export/home/cs130/cswan/pipeline130/trunk/sandbox/drmaa/helloworld.sh and check its status with qstat -j <number> The job runs perfectly (I made it so it touches the file "testfile" in the source directory). Then, using DRMAA, I submit an identical job. Running qstat -j on this job gives me identical information to the qsub version, but the script does not get executed. No idea what's going on. Probably won't be able to look at it any more tonight though. -Chris -------------------------------------------------- From: "Nils Homer" <nil...@uc...> Sent: Wednesday, May 13, 2009 5:56 PM To: <pip...@li...> Subject: [Pipeline130-developers] DRMAA > Just to let everyone know, I am still testing out the DRMAA interface via > the drmaa example in the sandbox folder (note: there are hard-coded > links). > Anyhow, if someone figures out how to get a job to successfully run (mine > are always aborted after submission), let me know. > > Nis > > > > ------------------------------------------------------------------------------ > The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your > production scanning environment may not be a perfect world - but thanks to > Kodak, there's a perfect scanner to get the job done! With the NEW KODAK > i700 > Series Scanner you'll get full speed at 300 dpi even with all image > processing features enabled. http://p.sf.net/sfu/kodak-com > _______________________________________________ > Pipeline130-developers mailing list > Pip...@li... > https://lists.sourceforge.net/lists/listinfo/pipeline130-developers > |
6 messages has been excluded from this view by a project administrator.
