Using scheduler parameters within other scheduler parameters (via substitution)

[Zach]

Summary of problem:
In a standard copy job, I would like to create a target directory a la

/my/dir/YYYY/MM/DD,

where the date fields are NOT populated from the current day. As far as I know, the standard job replacements via [date: ***] only handle the current date. My proposed solution is to store the YYYY, MM, and DD values in an external parameters file:

day_params.xml:

<?xml version="1.0" encoding="ISO-8859-1"?>
<params>
  <param name="year" value="2013"/>
  <param name="month" value="12"/>
  <param name="day" value="11"/>
</params>

and then to use those in the parameters file for the sos.scheduler.file.JobSchedulerCopyFile standard job by including another external parameters file:

file_transfer_params.xml:

<?xml version="1.0" encoding="ISO-8859-1"?>
<params>
  <param name="target_file" value="/my/dir/$SCHEDULER_PARAM_YEAR/$SCHEDULER_PARAM_MONTH/$SCHEDULER_PARAM_DAY"/>
</params>

Those two do need to be separate, currently. I have tried "including" both of those xml files in the same job, including day_params in one job and using it to set >> $SCHEDULER_RETURN_VALUES, and also tried many combinations of \$JSPAR_YEAR, $year, \$year, \${year}, etc formats I've found peppered through various similar requests I've found on-line, but to no avail.

Any suggestions?? This is a pretty standard concept, right, so how is this usually dealt with?

[Zach]

Any ideas, gang? Or is there perhaps a better place for me to ask this question?

[soskb]

Hi

have a look at this example:

<job title="Copies one or more files" order="no" name="CopyFilewithParameterSubstitution">
<settings>
<log_level><![CDATA[debug9]]></log_level>
</settings>

<description >
    <include  file="jobs/JobSchedulerCopyFile.xml"/>
</description>

<params >
    <param  name="create_dir" value="true"/>

    <param  name="target_file" value="c:/temp/%year%/%month%/%day%/"/>

    <param  name="source_file" value="c:/temp"/>

    <param  name="file_spec" value=".*\.xml$"/>

    <param  name="day" value="08"/>

    <param  name="year" value="2014"/>

    <param  name="month" value="01"/>
</params>

<script  language="java" java_class="sos.scheduler.file.JobSchedulerCopyFile"/>

<run_time />

</job>

It works with an include-file as well. Another option is to use an order and a job chain and modify/create the parameters in the job (state) in one of the predecessors of the copy job.

hth

[Zach]

Thanks so much for getting back to me, soskb!

I have tried what you have done above and agree that it works, but it isn't exactly what I'm hoping to do.

What I'd like to do is declare these variables in only one job and then use them dynamically in other jobs (not in the hard-coded, imported XML file).

Several SOS documentation sites (e.g. this page) talk about the ability to use internal substitution like:

Job 1: add variables {year, month, date} to SCHEDULER_RETURN_PARAMS so that $SCHEDULER_PARAM_{YEAR,MONTH,DATE} are all available in the following jobs.
Job 2: declare a new parameter

datestring = $SCHEDULER_PARAM_YEAR/$SCHEDULER_PARAM_MONTH/$SCHEDULER_PARAM_DAY

To date I have never managed to get this to work in the regular case, even downloading the exact xml files on that website (and converting them to *nix), let alone the case I actually want.

Several other pages (e.g. this one) talk about the ability to load parameters from external files. What would be ideal, for my situation, would be to dynamically replace pieces of the external XML file with the variables loaded in previous jobs. As it is, I can't do either part.

I'd really like to be able to do what I suggested here, but I don't know that your solution is the best way. If end up just rewriting 100s of XML files to solve the problem, I might as well go back to CSVs and python scripts.

Any more helpful hints / thoughts??

[soskb]

Hi Zach,

waht you want to do is:

1) create a separate xml-file with the parameter names and values:

2) use this file in any other job or order.

Solution is:

1) create, e.g. by you application or a script, the file like this:

<params> <param name="year" value="2014"/> <param name="month" value="03"/> <param name="day" value="06"/> </params>

Save this file with a unique file-name: current-booking-date.params.xml in the live-folder.

2) use the include tag in the parameter definition of the job/order to include the content of this file in the parameters of the job/order.

<job>
<params>
<include live_file="current-booking-date.params.xml"/>
<param name="target_file" value="c:/temp/%year%/%month%/%day%/"/> </params>

<script  language="shell">
    <![CDATA[

ping -n 10 localhost
]]>
</script>

</job>

Every time you recreate this file, JS will refresh the objects, which are using this file.

btw: you should use \${year} instead of \${SCHERDULER_PARAM_YEAR} because the prefix "SCHEDULER_PARAM" is used by JS when creating environment variables only.

[Uwe Risse]

What JobScheduler version do you use?

[Zach]

The above was the case for JS version 1.5.3253 (64 bit). I recently upgraded to version 1.6.4043 (64 bit) and am seeing the same results.

[Uwe Risse]

I have create two jobs.

One init job, that sets the value for param data to myDate

<job  order="yes">
    <script  language="shell">
        <![CDATA[
echo date=mydate >> $SCHEDULER_RETURN_VALUES
        ]]>
    </script>

    <run_time />
</job>

One execute job, that echos the value

<job  order="yes">
    <script  language="shell">
        <![CDATA[
echo value is  $SCHEDULER_PARAM_DATE
        ]]>
    </script>

    <run_time />
</job>

and a job chain that put these both jobs together.

Is it this what you want to have?

<job_chain  orders_recoverable="yes" visible="yes" name="job_chain1">
    <job_chain_node  state="100" job="job_init" next_state="200" error_state="error"/>

    <job_chain_node  state="200" job="job_execute" next_state="success"     error_state="error"/>

    <job_chain_node  state="success"/>

    <job_chain_node  state="error"/>
</job_chain>

[Zach]

That is not quite what I want.

Here's a pretty basic example: Say I want to check for the existence of a file that has a name "YYYYmmdd.txt", but the date is yesterday's date (e.g. today it is February 27th, the file's name would be "20141226.txt"). The built-in java jobs for checking the existence of files requires a "file" parameter be set. My goal is to get that file name -- something which must be generated dynamically every day -- set as the value of the "file" parameter.

Some SOS links claim that if I create a parameter called "myfile" in one job and pass it to the next via SCHEDULER_RETURN_PARAMS that I can do the following:

:xml
<param name="file" value="${SCHEDULER_PARAM_MYFILE}">

This simply doesn't work, even in the most basic of cases.

Once I was able to get that to work, there is a second step -- trying to have one global location of the variables I am using to construct the file name (that is, yesterday's date pieces) so that I don't have to repeat this step many times. But let's do first things first!

Zach

[Uwe Risse]

The first trick is to use backslash in the parameter value.
The second one is to add the configuration_monitor.
To avoid error message like (can be ignored)

"java.io.FileNotFoundException: C:\Users\ur\Documents\sos-berlin.com\jobscheduler\scheduler_current\config\live\paramtest\job_chain1.config.xml "

add a dummy_parameter for a node in the job chain using JOE.

<job  title="check wether a file exist" order="yes" stop_on_error="no" name="FileExist">

    <params >
        <param  name="file" value="\${actual_date}.txt"/>
    </params>

    <script  language="java" java_class="sos.scheduler.file.JobSchedulerExistsFile"/>

    <monitor  name="configuration_monitor" ordering="0">
        <script  language="java" java_class_path=""     java_class="sos.scheduler.managed.configuration.ConfigurationOrderMonitor"/>
    </monitor>

    <run_time />
</job>

Now coming to the central setting of the value for the actual_date.
Add a job set_date with runtime once=yes (to let the job start when JobScheduler starts) and a single start.

<job  name="set_date">
    <params />

    <script  language="javax.script:rhino">
        <![CDATA[
function spooler_process(){

   var today = new Date();
   var day = today.getDate();
   var year = today.getFullYear();
   var month = today.getMonth()+1;

   actual_date = year + fill(month) + fill(day);

   spooler.variables().set_var("actual_date",actual_date);

    return false;
}

function fill(val){
    s = val+"";
    if (s.length == 1){
       return "0"+s;
    }else{
       return s;
    }
}
        ]]>
    </script>

    <run_time  once="yes">
        <period  single_start="00:00:01"/>
    </run_time>
</job>