Re: [Taskforest-discuss] Taskforest-discuss Digest, Vol 2, Issue 3

[Aijaz A. <aij...@gm...>]

Lana:

The second instance of J4() in the family isn't a call to J4.  Technically,
neither is the first.  Don't think of J1() and J4() as calls (like function
calls in code) but instead think of them as nodes in a graph.  In the family
below J4() will *never* run unless J1() is run successfully.  J1(),  in turn
will *never* run unless both J2() and J3() have both run.  Since J3() won't
run before 3:00, J4 will not run until some time after 3:00.  J4 will not
kick off.  The order of the sections (separated by dashed lines) does not
imply an order of execution.  The sections are only there to indicate that
these jobs on the next line don't depend on the jobs on the previous line
because of the line of dashes.

No matter how long J3 runs, J4 will only run once, and only after J1 (and
therefore only after J3) has run.

I don't know if I'm explaining this well..  Please let me know if any of
this doesn't make sense.

Thanks,

Aijaz.


On Tue, Apr 28, 2009 at 5:02 PM, Svetlana Lemeshov <lem...@mi...> wrote:

> Thank you Aijaz!
>
> This is a good news about multiple calls to a job within a family!
>
> However, I see a potentially problematic situation with your example.
> I stumbled on it while trying to model our test Family. Here is an example:
>
> Again, let's look at job J4 that shows up twice.
>
> start => '00:00', tz => 'America/Chicago', days =>
> 'Mon,Tue,Wed,Thu,Fri,Sat,Sun'
>
> J2()  J3(start=>'3:00')
>  J1()
>  J4()
> J12()   J13()
>
> --------------------------------------------------------------------------------
>  J6()    J7()
>    J5()
>
> --------------------------------------------------------------------------------
>  J5()    J4()
>    J8()
>
> --------------------------------------------------------------------------------
>
> For example J3(), is an extraction job that populates input file for J4()
> and can't run before certain time (ex.3:00AM). It will be sitting
> waiting for an
> input file, while the second call to J4() will actually kick in at 00:00
> without an input file and die with exit 1 right away. It may cause an
> undesirable effect.
>
> Actually, even if J3() does not have a specific start time, we still can
> run
> into a messy situation...
> When in your example J4() is called for the second time just to have its
> status checked, you ASSUME that it has already 'ran'.
> BUT if prerequisite jobs (J2, J3 and J1) are happen to be LONG running
> jobs....
> then the second ('status checking') occurrence of J4() will trigger the
> execution BEFORE the first occurrence!
>
> I am sure that this can somehow be handled. But these are kind of little
> 'challenges' that I am stumbling on while working on a test scenario
> involving
> 5-6 jobs. Given that our crontab file currently has over 1000
> entries... I think
> I will be done in a couple of years :)
>
> Thank you very much for all your help!
>
> Talk to you soon!
>
> Lana
>
>
>
> > Quoting tas...@li...:
>
> > Send Taskforest-discuss mailing list submissions to
> >       tas...@li...
> >
> > To subscribe or unsubscribe via the World Wide Web, visit
> >       https://lists.sourceforge.net/lists/listinfo/taskforest-discuss
> > or, via email, send a message with subject or body 'help' to
> >       tas...@li...
> >
> > You can reach the person managing the list at
> >       tas...@li...
> >
> > When replying, please edit your Subject line so it is more specific
> > than "Re: Contents of Taskforest-discuss digest..."
> >
> >
> > Today's Topics:
> >
> >   1. Fwd:  Looking for TaskForest USERS!!! (Aijaz Ansari)
> >   2. TaskForest 1.24 to be released this weekend. (Aijaz Ansari)
> >   3. TaskForest 1.24 is now available (Aijaz Ansari)
> >   4. Some additional questions about TaskForest (Svetlana Lemeshov)
> >   5. Re: Some additional questions about TaskForest (Aijaz Ansari)
> >
> >
> > ----------------------------------------------------------------------
> >
> > Message: 1
> > Date: Sat, 18 Apr 2009 16:45:02 -0500
> > From: Aijaz Ansari <en...@us...>
> > Subject: [Taskforest-discuss] Fwd:  Looking for TaskForest USERS!!!
> > To: tas...@li...
> > Message-ID:
> >       <dc3...@ma...>
> > Content-Type: text/plain; charset="iso-8859-1"
> >
> > Lana:
> >
> > Some more comments about calendars and Family dependencies:
> >
> > The next major release of TaskForest after the one currently in
> development
> > (Tokens) will support Calendars.  You'll be able to associate a Family
> with
> > a calendar date file, which is essentially a list of all dates in the
> year
> > on which this Family will run, or you could use expressions that evaluate
> to
> > one or more dates. For example:
> >
> >   1. Last Friday of every month
> >   2. Every July 4th, etc.
> >
> > You could also apply binary operators like AND, OR, and EXCLUSIVE OR to
> > calendars to come up with custom calendars.
> >
> > I expect this version to come out in May, 2009.
> >
> > Now for dependencies between families:
> >
> > I think I can do what you described by including some syntax that refers
> to
> > jobs in other families.  If Family F1 has job J1, and family F2 has job
> J2,
> > we could make J2 depend on J1 using syntax similar to this in F2:
> >
> > F1::J1()
> > J2() # J2 will only run after J1 in F1 runs.
> >
> > This is not supported yet, but it should be pretty easy to implement.
>  It's
> > a good idea and I'll let you know what I find out in a day or two.
> >
> > Thanks,
> >
> > Aijaz.
> > -------------- next part --------------
> > An HTML attachment was scrubbed...
> >
> > ------------------------------
> >
> > Message: 2
> > Date: Thu, 23 Apr 2009 09:48:20 -0500
> > From: Aijaz Ansari <en...@us...>
> > Subject: [Taskforest-discuss] TaskForest 1.24 to be released this
> >       weekend.
> > To: tas...@li...
> > Message-ID:
> >       <dc3...@ma...>
> > Content-Type: text/plain; charset="iso-8859-1"
> >
> > Hello:
> >
> > I just wanted to let you all know that the latest version of TaskForest
> will
> > be released this Sunday.  It includes support for tokens as well as
> better
> > support for families with foreign time zones.  I will spend the next few
> > days running it on my prod boxes and completing the documentation before
> the
> > release.
> >
> > Thanks,
> >
> > Aijaz.
> > -------------- next part --------------
> > An HTML attachment was scrubbed...
> >
> > ------------------------------
> >
> > Message: 3
> > Date: Sun, 26 Apr 2009 23:49:06 -0500
> > From: Aijaz Ansari <en...@us...>
> > Subject: [Taskforest-discuss] TaskForest 1.24 is now available
> > To: tas...@li...,
> >       tas...@li...
> > Message-ID:
> >       <dc3...@ma...>
> > Content-Type: text/plain; charset="iso-8859-1"
> >
> > TaskForest version 1.24 is now available.  Please visit
> > http://www.taskforest.com for the most recent documentation as well as
> > instructions on how to download the software.
> >
> > This release introduces the concept of tokens.  A token is a new kind of
> > dependency.  If a job depends on a token, it must acquire a free token
> > before it can run, or wait for another job to release a token after
> > completion.  Since there is a finite set of tokens, you can use tokens to
> > prevent too many jobs from running at the same time.
> >
> > For example, if there are 20 jobs that rely on token T, but only 5
> instances
> > of the token, only a maximum of 5 of those jobs may run at the same time.
> >
> > This release also formalizes the way Families with foreign time zones are
> > handled.  This allows you to 'time shift' your family, essentially
> allowing
> > your family to run for 24 hours in a later timezone.  This shifts the end
> of
> > day towads the end of the family's running time.
> >
> > Thanks,
> >
> > Aijaz
> > -------------- next part --------------
> > An HTML attachment was scrubbed...
> >
> > ------------------------------
> >
> > Message: 4
> > Date: Tue, 28 Apr 2009 14:34:48 -0400
> > From: Svetlana Lemeshov <lem...@MI...>
> > Subject: [Taskforest-discuss] Some additional questions about
> >       TaskForest
> > To: tas...@li...
> > Message-ID: <200...@we...>
> > Content-Type: text/plain;     charset=ISO-8859-1
> >
> > Hello Aijaz,
> >
> > We are currently using crontab and we face some issues related to the
> > daylight
> > time saving change:
> > - in one case some jobs may run twice (scheduled to run between 2AM and
> 3AM)
> > - while in another they do not run at all.
> >
> > We manually take care of these situations.
> >
> > So, I wonder how TaskForest handles time changes with GMT option.
> >
> > Aijaz, can you please give me some information about implementation of
> > the Calendars in the future releases of the TaskForest?
> >
> > When do you anticipate it to be in production?
> >
> > Here are some questions (if you don't mind):
> > - Is there a way to turn off 'Days of the week' from the Family all
> together?
> > I am contemplating to run TaskForest from the crontab to have more
> > control and
> > flexibility with the schedule.
> >
> > - What happens when the same job gets called within a family multiple
> > times? I
> > DO understand that it will get executed only once, but after that, can I
> keep
> > calling the same job from different parts of the Family just to check its
> > status? I can see it to be useful in cases when a chain of dependant jobs
> > can't be easily layed out with a straightforward top down design.
> >
> > It is probably confusing and I should send you an example to illustrate
> > my question.
> >
> > I spent some time setting a simple test family, but I feel that
> > its current layout still does not reflect what I want it to do.
> >
> > I find that a single example of the 'Forest' in TaskForest documentation
> > is too basic to demonstrate how to handle even a little more complex
> > dependencies.
> >
> >
> > Thank you very much!
> >
> > Lana
> >
> >
> >
> > ------------------------------
> >
> > Message: 5
> > Date: Tue, 28 Apr 2009 15:02:11 -0500
> > From: Aijaz Ansari <en...@us...>
> > Subject: Re: [Taskforest-discuss] Some additional questions about
> >       TaskForest
> > To: tas...@li...
> > Message-ID:
> >       <dc3...@ma...>
> > Content-Type: text/plain; charset="iso-8859-1"
> >
> > Lana:
> >
> > GMT ISSUES
> > As of version 1.24, there's an undocumented way to fool taskforest into
> > thinking the current time is whatever you tell it.  I use it for testing.
> > I'll test out your GMT questions, but I'm almost certain that GMT is
> immune
> > to any DST issues, since it doesn't have DST.  This also means that
> you'll
> > have exactly 24 hours in every day, and not 23 hours 1 day a year and 25
> > another day.  I'm 99% sure, but I'll run tests and let you know.  Thanks
> for
> > bringing this up.  This should be a test.
> >
> > CALENDARS
> > The calendar code is currently in Dev.  I hope to have it released in
> > version 1.25 on May 17.  Because I want the code to be backward
> compatible
> > with older family files, Family files will accept either the current
> "days
> > => 'Mon,Tue'" format or the new "calendar => CAL_NAME" format.
> > CAL_NAME is a name of a calendar that will be defined in the config file.
> > A calendar is basically a set of instructions that include or exclude
> days.
> > Here are some examples:
> >
> > Example 1)
> > + 2009/04/24
> >
> > Familes that use this calendar will run only once. On Apr 24, 2009 only.
> >
> ----------------------------------------------------------------------------------------------------------
> > Example 2)
> > + 2009/*/*
> > - 2009/01/01
> >
> > Families that use this calendar will run every day in 2009, except for
> New
> > Years Day.
> >
> ------------------------------------------------------------------------------------------------------------
> > Example 3)
> > + */*/*
> > - fourth Thu */11
> >
> > Families that use this calendar will run every day of every month of
> every
> > year, except for the fourth Thursday of November of each year (U.S.
> > Thanksgiving day)
> >
> ---------------------------------------------------------------------------------------------------------------
> > Example 4)
> > + second last Sunday */*/*
> >
> > Families that use this calendar will run on the second-to-last Sunday of
> > every month.
> >
> > Using calendars like this might be the better than relying on cron's
> > scheduling. It depends on what exactly you're trying to do.  If you have
> > specific scheduling needs, let me know, and I'll tell you if and how you
> > could configure taskforest to meet that schedule.
> >
> > Again, I expect this to be in version 1.25 in mid-May.
> >
> > MULTIPLE INSTANCES OF JOBS IN THE FAMILY FILE
> > ----------------------------------------------------------------------
> > Yes, you can have multiple instances of jobs in a family file, to check
> on
> > status.  There are certain cases, where you *cannot* properly model
> > dependencies unless you have multiple instances like this.  One of the
> test
> > Family files that's included in the distribution (the one wrongly named
> > "t/family_archive/SIMPLE") has this scenario.  Look at job J4.  It shows
> up
> > twice. I'm copying the Family file here for your reference:
> >
> > start => '00:00', tz => 'America/Chicago', days =>
> > 'Mon,Tue,Wed,Thu,Fri,Sat,Sun'
> > J2()  J3()
> >  J1()
> >  J4()
> > J12()   J13()
> >
> --------------------------------------------------------------------------------
> >  J6()    J7()
> >    J5()
> >
> --------------------------------------------------------------------------------
> >  J5()    J4()
> >    J8()
> >
> --------------------------------------------------------------------------------
> > J9()
> > J10()
> > J11()
> >
> > So yes, you can, and in many cases should repeat job names in the family
> > file.   I hope I've answered your questions well.  Pls let me know if you
> > have any other questions.
> >
> > Thanks,
> >
> > Aijaz.
> >
> >
> > On Tue, Apr 28, 2009 at 1:34 PM, Svetlana Lemeshov <lem...@mi...>
> wrote:
> >
> >> Hello Aijaz,
> >>
> >> ...
> >> So, I wonder how TaskForest handles time changes with GMT option.
> >>
> >> Aijaz, can you please give me some information about implementation of
> >> the Calendars in the future releases of the TaskForest?
> >>
> >> When do you anticipate it to be in production?
> >>
> >> Here are some questions (if you don't mind):
> >> - Is there a way to turn off 'Days of the week' from the Family all
> >> together?
> >> I am contemplating to run TaskForest from the crontab to have more
> control
> >> and
> >> flexibility with the schedule.
> >>
> >> - What happens when the same job gets called within a family multiple
> >> times? I
> >> DO understand that it will get executed only once, but after that, can I
> >> keep
> >> calling the same job from different parts of the Family just to check
> its
> >> status? I can see it to be useful in cases when a chain of dependant
> jobs
> >> can't be easily layed out with a straightforward top down design.
> >>
> >> ...
> >>
> >>
> > -------------- next part --------------
> > An HTML attachment was scrubbed...
> >
> > ------------------------------
> >
> >
> ------------------------------------------------------------------------------
> > Register Now & Save for Velocity, the Web Performance & Operations
> > Conference from O'Reilly Media. Velocity features a full day of
> > expert-led, hands-on workshops and two days of sessions from industry
> > leaders in dedicated Performance & Operations tracks. Use code vel09scf
> > and Save an extra 15% before 5/3. http://p.sf.net/sfu/velocityconf
> >
> > ------------------------------
> >
> > _______________________________________________
> > Taskforest-discuss mailing list
> > Tas...@li...
> > https://lists.sourceforge.net/lists/listinfo/taskforest-discuss
> >
> >
> > End of Taskforest-discuss Digest, Vol 2, Issue 3
> > ************************************************
> >
>
>
>
>
> ------------------------------------------------------------------------------
> Register Now & Save for Velocity, the Web Performance & Operations
> Conference from O'Reilly Media. Velocity features a full day of
> expert-led, hands-on workshops and two days of sessions from industry
> leaders in dedicated Performance & Operations tracks. Use code vel09scf
> and Save an extra 15% before 5/3. http://p.sf.net/sfu/velocityconf
> _______________________________________________
> Taskforest-discuss mailing list
> Tas...@li...
> https://lists.sourceforge.net/lists/listinfo/taskforest-discuss
>