[Exist-development] new functionality: Execution Pipeline

[Thomas W. <tho...@gm...>]

I would like to propose a new functionality that I believe could be very
beneficial for eXist users:
Asynchronous Execution PipelineThis a mechanism for execution of number of
asynchronous jobs simultaneously.  It is very useful for executing long
running jobs or in cases where it is impossible to predict how long it will
take to perform the operation. Every job will run as a separated thread and
the jobID and the estimated delay will be returned immediately to the
caller.

Use cases: 1. Executing long running queries

   - Callback function will be used to store the result, at a location
   according to the function-parameters.
   - A client checking periodically the status of this job will take next
   action.

2. Fetching data from (large) number of remote URLs

   - An XQuery or a scheduled job creates XX execution pipeline entries for
   each remote server.
   - Callback functions are used to store the results, at a location
   according to the function-parameters.
   - The batch callback function will combine the result and trigger the
   next action.

3. Federated search, on a web client

   - A web client sends a search request to a local XQuery, that creates XX
   execution pipeline entries for each remote server and returns to the web
   client a batch-id.
   - The web client queries the status for the jobs with this batch-id
   periodically and when some of the jobs has status 'completed', web client
   gets the result for this job and displays it on the screen asynchronously.

4. Federated search, on a the server

   - A web client sends a search request to a local XQuery, that creates XX
   execution pipeline entries for each remote server and returns to the web
   client a batch-id.
   - Every job callback function will save the result at a location
   according to the function-parameters. The batch callback function will
   combine the result.
   - The web client queries the status for this batch periodically and when
   the batch is completed, web client gets the result and displays combined
   result set on the screen asynchronously.


*5. Data Replication *

   - An XQuery or a scheduled job creates XX execution pipeline entries for
   each remote server.
   - Execute-before function will identify what needs to be replicated.
   - The main function does the replication.
   - The batch callback function moves the replication marker.



A call to the Execution Pipe Line:
   *execution-pipeline:addJob*( function, function-parameters,
pipeline-parameters )*
 *returning :
    handlerID, estimated-delay,  function-parameters


To get the result we need to call another function:
    *execution-pipeline:getJobResults*( handlerID, autoClose )
returning either:
   * the result data set*. if autoClose is true then close the job and
release all used resources.
or
   same handlerID, new-estimated-delay,function -parameters
or
   unknown-handlerID error

*execution-pipeline:getJobStatus*( handlerID )
returns
        status of the job, function-parameters for this job

*execution-pipeline:getBatchStatus*(  batch-ID )
returns
        the status for all jobs from a particular batch ID.


*execution-pipeline:getStatus*(  )
returns
        the status for all jobs.


*execution-pipeline:closeJob*( handlerID )
*execution-pipeline:closeBatch*( batchID )
*execution-pipeline:closeAll*( )


*function-parameters:*

   - job-statistic-id: used to keep average time for execution of this
   function. average time= (previous-average-time + last-execution-time)/2.
   URL with specific parameters could be used as an ID.
   - execute-before function: when provided, it will be called before
   calling the main function for this job. If the result is 0 then proceed with
   the main function, otherwise use the result as number of milliseconds to
   put this job to sleep and try later.
   - callback function: when provided callback-function will be called as
   callback-function( handlerID, result, function-parameters ). if it
   returns true() the job will be closed.
   - any other parameters that may be used by the callback function.


*pipeline-parameters:*

   - batch-ID - to group
   - batch-callback-function: called when all jobs from the batch are
   completed.
   - any other parameters that may be used by the callback function.


Any comments?

Thomas


------

Thomas White

Mobile:+44 7711 922 966
Skype: thomaswhite
gTalk: thomas.0007
Linked-In:http://www.linkedin.com/in/thomaswhite0007
facebook: http://www.facebook.com/thomas.0007