Re: [Exist-development] new functionality: Execution Pipeline

[Adam R. <ad...@ex...>]

Can you not already do most (if not all) of this by Scheduling XQuery
jobs with eXist's Scheduler?


2010/1/25 Thomas White <tho...@gm...>:
> I would like to propose a new functionality that I believe could be very
> beneficial for eXist users:
>
> Asynchronous Execution Pipeline
>
> This a mechanism for execution of number of asynchronous jobs
> simultaneously.  It is very useful for executing long running jobs or in
> cases where it is impossible to predict how long it will take to perform the
> operation. Every job will run as a separated thread and the jobID and the
> estimated delay will be returned immediately to the caller.
>
> Use cases:
>
> 1. Executing long running queries
>
> Callback function will be used to store the result, at a location according
> to the function-parameters.
> A client checking periodically the status of this job will take next action.
>
> 2. Fetching data from (large) number of remote URLs
>
> An XQuery or a scheduled job creates XX execution pipeline entries for each
> remote server.
> Callback functions are used to store the results, at a location according to
> the function-parameters.
> The batch callback function will combine the result and trigger the next
> action.
>
> 3. Federated search, on a web client
>
> A web client sends a search request to a local XQuery, that creates XX
> execution pipeline entries for each remote server and returns to the web
> client a batch-id.
> The web client queries the status for the jobs with this batch-id
> periodically and when some of the jobs has status 'completed', web client
> gets the result for this job and displays it on the screen asynchronously.
>
> 4. Federated search, on a the server
>
> A web client sends a search request to a local XQuery, that creates XX
> execution pipeline entries for each remote server and returns to the web
> client a batch-id.
> Every job callback function will save the result at a location according to
> the function-parameters. The batch callback function will combine the
> result.
> The web client queries the status for this batch periodically and when the
> batch is completed, web client gets the result and displays combined result
> set on the screen asynchronously.
>
> 5. Data Replication
>
> An XQuery or a scheduled job creates XX execution pipeline entries for each
> remote server.
> Execute-before function will identify what needs to be replicated.
> The main function does the replication.
> The batch callback function moves the replication marker.
>
> A call to the Execution Pipe Line:
>    execution-pipeline:addJob( function, function-parameters,
> pipeline-parameters )
>  returning :
>     handlerID, estimated-delay,  function-parameters
>
>
> To get the result we need to call another function:
>     execution-pipeline:getJobResults( handlerID, autoClose )
> returning either:
>     the result data set. if autoClose is true then close the job and release
> all used resources.
> or
>    same handlerID, new-estimated-delay,function -parameters
> or
>    unknown-handlerID error
>
> execution-pipeline:getJobStatus( handlerID )
> returns
>         status of the job, function-parameters for this job
>
> execution-pipeline:getBatchStatus(  batch-ID )
> returns
>         the status for all jobs from a particular batch ID.
>
>
> execution-pipeline:getStatus(  )
> returns
>         the status for all jobs.
>
>
> execution-pipeline:closeJob( handlerID )
> execution-pipeline:closeBatch( batchID )
> execution-pipeline:closeAll( )
>
>
> function-parameters:
>
> job-statistic-id: used to keep average time for execution of this function.
> average time= (previous-average-time + last-execution-time)/2. URL with
> specific parameters could be used as an ID.
> execute-before function: when provided, it will be called before calling the
> main function for this job. If the result is 0 then proceed with the main
> function, otherwise use the result as number of milliseconds to put this job
> to sleep and try later.
> callback function: when provided callback-function will be called as
> callback-function( handlerID, result, function-parameters ). if it returns
> true() the job will be closed.
> any other parameters that may be used by the callback function.
>
> pipeline-parameters:
>
> batch-ID - to group
> batch-callback-function: called when all jobs from the batch are completed.
> any other parameters that may be used by the callback function.
>
> Any comments?
>
> Thomas
>
>
> ------
>
> Thomas White
>
> Mobile:+44 7711 922 966
> Skype: thomaswhite
> gTalk: thomas.0007
> Linked-In:http://www.linkedin.com/in/thomaswhite0007
> facebook: http://www.facebook.com/thomas.0007
>
> ------------------------------------------------------------------------------
> Throughout its 18-year history, RSA Conference consistently attracts the
> world's best and brightest in the field, creating opportunities for
> Conference
> attendees to learn about information security's most important issues
> through
> interactions with peers, luminaries and emerging and established companies.
> http://p.sf.net/sfu/rsaconf-dev2dev
> _______________________________________________
> Exist-development mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-development
>
>



-- 
Adam Retter

eXist Developer
{ United Kingdom }
ad...@ex...
irc://irc.freenode.net/existdb