Sequential proceeding past INCOMPLETE

[Bob A.]

Hi Rob,

Would you please tell me how you would accomplish the following:

A: Sequential job
  - B: Nested Oddjob (that should reflect the lowest state of its children)
      - B.1: Exec job (which might reach COMPLETE or INCOMPLETE/EXCEPTION)
      - B.2: State:if job that runs every single time that B.1 reaches a state of COMPLETE/INCOMPLETE/EXCEPTION (B.2 practically always reaches COMPLETE because it is the logging step)
  - C: Nested Oddjob

So for example, let's say that we start A.  Then B starts.  If B.1 completes, then B.2 should run, and A should be marked as COMPLETE once B.2 reaches COMPLETE (which B.2 always does). 

BUT, lets say that B.1 reaches INCOMPLETE/EXCEPTION.  Then B.2 should still start, and A should reflect the state of B.1.  Then if we soft reset A, B.1 should be reset to READY and B.2 should also be reset to READY even though it was marked as COMPLETE. 

Most importantly, I don't want C to start unless B gets marked COMPLETE.

I thought that I had something that should work, but it kept running C even though B ended as INCOMPLETE.

Here's what I had:

Parent-oddjob
  - A: Sequential job
    - B: Nested oddjob
      - Parallel job
        - B.1: Exec
        - B.3: State:or
          - State:equals COMPLETE
            - State:mirror (B.1)
          - State:equals INCOMPLETE
            - State:mirror (B.1)
          - State:equals EXCEPTION
            - State:mirror (B.1)
        - State:resets HARDEN
          - Scheduling:trigger (B.3)
            - B.2: State:if
    - C: Nested oddjob

With my attempt, C runs even when B.1 reaches INCOMPLETE.

Thanks for your input,
   Bob

[Bob A.]

I believe that the problem has something to do with the Scheduling:trigger.  If I replace that step with an echo-job, then C will not start until B is COMPLETE (which is the behavior I expected/wanted).

[Bob A.]

Workaround: I have replaced the Scheduling:trigger (that contained B.2) with a SequentialJob containing a WaitJob and then B.2.  The WaitJob waits until B.3 is COMPLETE and then B.2 starts.  Most importantly, C does not begin until B is marked COMPLETE.

I still can't figure out why the scheduling:trigger was causing the next nested oddjob to start (regardless of the state of B). 

[Bob A.]

Just for fun, I tried replacing the scheduling:trigger with a state:cascade.  Same problem: C starts before B is even marked INCOMPLETE.

[Rob Gordon]

This is the confusing difference between jobs which execute when the executing thread reaches them, and jobs which execute asynchronously on an event. The executing thread only activiates the trigger, it doesn't wait. Cascade should work, moving everything to be event driven - but the executing thread only starts the cascade, it doesn't wait. Your outer sequential would also need to be a cascade. state:join will cause the executing thread to wait for a state event.

However, trigger is for reacting to asychronous state events (such as from a scheduled job), exec is synchronous, so you don't need the trigger. Also because exec isn't asynchronous it's guaranteed to be in one of the complete/incomplete/exception states before the thread of execution arrives at the next job - so you don't need the 'or' either. What's the 'if' executing? You can probably put this job straight after the exec and get the desired result.

If been considering introducing an 'active' state to differentiate betweem those jobs that still have the executing thread, and those that don't but have been started. I think this would be good in this situation.

Hope this helps.

Rob.

[Bob A.]

Thanks for responding.  It might take me a bit to absorb the asynchronous part.

Here's what I want to do:
1. set variables
2. run an exec job
3. write to logfile regardless of which state the exec job ends in
  3a. if logfile exists then append to it.
  3b. if logfile does not exist then create it.

I think the following should work. 

  - Nested Oddjob
    - Parallel
      - Sequential
        - Variables
        - Execute
      - Sequential
        - State:or
          - State:equals COMPLETE
            - State:mirror (Execute)
          - State:equals INCOMPLETE
            - State:mirror (Execute)
          - State:equals EXCEPTION
            - State:mirror (Execute)
        - State:if
          - Exists (logfile)
          - Append to logfile
          - Create logfile

Narrative: the parallel starts both sequentials.  The first sequential runs the variables job and the exec job.  The second sequential has a state:or job which becomes COMPLETE when the exec job reaches a state of COMPLETE or INCOMPLETE or EXCEPTION.  The second sequential then proceeds to process the state:if job which writes to the logfile.

Here's the weirdness: this solution runs GREAT about 1/4 of the time.  But usually, the second sequential only runs the state:or and does NOT proceed to the state:if.  I tried just starting then hard resetting over and over again.  And sometimes the state:if runs, but most of the time it doesn't.

Brainstorming: Could this have to do with -Doddjob.executors.default.poolsize?  Does state:or continue to process even after it is marked COMPLETE, and maybe that holds onto a thread?

Here's how i run oddjob if that matters:
start "Oddjob Clients" "%java_home%\bin\javaw.exe" -Doddjob.executors.default.poolsize=40 -jar C:\PDI\Oddjob\run-oddjob.jar -f "\Oddjob\Servers\explorer_clients.xml"

Thanks for helping me understand this,
   Bob

[Rob Gordon]

I think the inconsistency is a race condition between the trigger and the outer sequential job. The trigger doesn't fire in time, so sequential doesn't see the incomplete in time.

Here's a way I think you can achieve what you want:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<oddjob>
    <job>
        <sequential>
            <jobs>
                <state:if name="Always Completes" xmlns:state="http://rgordon.co.uk/oddjob/state">
                    <jobs>
                        <state:flag id="main-job" state="INCOMPLETE"/>
                    </jobs>
                </state:if>
                <echo text="This will always run"/>
                <state:mirror job="${main-job}" name="Reflect state for sequential to pick up on." xmlns:state="http://rgordon.co.uk/oddjob/state"/>
            </jobs>
        </sequential>
    </job>
</oddjob>

It uses a side affect of the state:if that if there is no then/else, it always completes. Your log writing logic would replace the echo job.

There is a bug with mirror job that's causing an error to be logged on oddjob shutdown - but it doesn't affect the functionality.

Hope this helps.
Rob.