Menu
▾
▴
exist-development
|
From: Andrzej J. T. <an...@ch...> - 2010-02-02 16:15:23
|
Thomas asked: >> do believe, having asynchronous mechanism for fetching data in eXist will >> > give us foundation for many?interesting new?ideas. >> > I will need this functionality in about two months time and I?really hope >> > somebody from the dev team will be interested in implementing the execution >> > pipeline. To which Adam replied: > I think we will be very pushed for time to meet that deadline. The > roadmap for the next two versions is already laid out - although it is > subject to change. This feature, asynchronous execution of XQueries, is something that I'm really interested in having as well. And I'm willing to implement this within the next two months (probably sooner than that!). I was thinking of a new function along the lines of: util:eval-async( ... ) which would return some sort of id for the spawned xquery that could be used to check completion status and obtain the results of the execution. Same function signatures as the original util:eval() function to start. Would this kind of new function meet your needs as well, Thomas? Thoughts from the crew on such a new function? -- Andrzej Taramina Chaeron Corporation: Enterprise System Solutions http://www.chaeron.com |
|
From: Thomas W. <tho...@gm...> - 2010-02-02 19:44:21
|
Andrzej, util:eval-async will be close to what I need with one difference - the ability to create a batch of jobs. I believe it is important especially when we scale things up. If we have an application where every user can fire a federated search to 25 remote servers, we do need a simple and robust way to cancel incompleted long running jobs, if the user makes a new federated search, logs off prematurely or when the session times out. With batch functions we can combine results (with batch callback function) or get with one command all available data so far from all completed jobs. Thomas ------ Thomas White Mobile:+44 7711 922 966 Skype: thomaswhite gTalk: thomas.0007 Linked-In:http://www.linkedin.com/in/thomaswhite0007 facebook: http://www.facebook.com/thomas.0007 On 2 February 2010 16:15, Andrzej Jan Taramina <an...@ch...> wrote: > > Thomas asked: > > >> do believe, having asynchronous mechanism for fetching data in eXist will > >> > give us foundation for many?interesting new?ideas. > >> > I will need this functionality in about two months time and I?really hope > >> > somebody from the dev team will be interested in implementing the execution > >> > pipeline. > > To which Adam replied: > > > I think we will be very pushed for time to meet that deadline. The > > roadmap for the next two versions is already laid out - although it is > > subject to change. > > This feature, asynchronous execution of XQueries, is something that I'm really interested in having as well. > > And I'm willing to implement this within the next two months (probably sooner than that!). > > I was thinking of a new function along the lines of: > > util:eval-async( ... ) > > which would return some sort of id for the spawned xquery that could be used to check completion status and obtain the > results of the execution. Same function signatures as the original util:eval() function to start. > > Would this kind of new function meet your needs as well, Thomas? > > Thoughts from the crew on such a new function? > > > -- > Andrzej Taramina > Chaeron Corporation: Enterprise System Solutions > http://www.chaeron.com |
|
From: Adam R. <ad...@ex...> - 2010-02-03 11:18:47
|
> If we have an application where every user can fire a federated search > to 25 remote servers, we do need a simple and robust way to cancel > incompleted long running jobs, if the user makes a new federated > search, logs off prematurely or when the session times out. This is where I struggle to understand your vision a little bit, and maybe I need to revisit the original discussion. But all of what you want above can be achieved with the Scheduler module - the Scheduler module enables you to immediately Schedule an XQuery for one-shot asynchronous execution, it also manages the status of running Queries and enables you to cancel long running queries. > > > > ------ > > Thomas White > > Mobile:+44 7711 922 966 > Skype: thomaswhite > gTalk: thomas.0007 > Linked-In:http://www.linkedin.com/in/thomaswhite0007 > facebook: http://www.facebook.com/thomas.0007 > > > > On 2 February 2010 16:15, Andrzej Jan Taramina <an...@ch...> wrote: >> >> Thomas asked: >> >> >> do believe, having asynchronous mechanism for fetching data in eXist will >> >> > give us foundation for many?interesting new?ideas. >> >> > I will need this functionality in about two months time and I?really hope >> >> > somebody from the dev team will be interested in implementing the execution >> >> > pipeline. >> >> To which Adam replied: >> >> > I think we will be very pushed for time to meet that deadline. The >> > roadmap for the next two versions is already laid out - although it is >> > subject to change. >> >> This feature, asynchronous execution of XQueries, is something that I'm really interested in having as well. >> >> And I'm willing to implement this within the next two months (probably sooner than that!). >> >> I was thinking of a new function along the lines of: >> >> util:eval-async( ... ) >> >> which would return some sort of id for the spawned xquery that could be used to check completion status and obtain the >> results of the execution. Same function signatures as the original util:eval() function to start. >> >> Would this kind of new function meet your needs as well, Thomas? >> >> Thoughts from the crew on such a new function? >> >> >> -- >> Andrzej Taramina >> Chaeron Corporation: Enterprise System Solutions >> http://www.chaeron.com > -- Adam Retter eXist Developer { United Kingdom } ad...@ex... irc://irc.freenode.net/existdb |
|
From: Thomas W. <tho...@gm...> - 2010-02-03 15:47:06
|
Adam, Where shell I start, may be from the fact I was not aware there are new functions in the scheduler. Very lame :-) I guess when you say "one-shot asynchronous execution" you mean using schedule-xquery-periodic-job called with $period=0, and $repeat=1. It seams then I have everything I need to create the batch asynch functionality I described earlier. BTW, is there a way to reduce the number remaining calls for * schedule-xquery-periodic-job* in run time? Use case: I need to process a dataset running periodically a XQuery function that process a chunk of that data. I expect to have 5000 calls but the data set is smaller this time and I need to cancel it after the 50th call. Can I do this? If not, I hope we can then extend *schedule-xquery-periodic-job* function to cancel the job if the scheduled function returns false(). Regards, Thomas ------ Thomas White Mobile:+44 7711 922 966 Skype: thomaswhite gTalk: thomas.0007 Linked-In:http://www.linkedin.com/in/thomaswhite0007 facebook: http://www.facebook.com/thomas.0007 On 3 February 2010 11:18, Adam Retter <ad...@ex...> wrote: > > If we have an application where every user can fire a federated search > > to 25 remote servers, we do need a simple and robust way to cancel > > incompleted long running jobs, if the user makes a new federated > > search, logs off prematurely or when the session times out. > > This is where I struggle to understand your vision a little bit, and > maybe I need to revisit the original discussion. But all of what you > want above can be achieved with the Scheduler module - the Scheduler > module enables you to immediately Schedule an XQuery for one-shot > asynchronous execution, it also manages the status of running Queries > and enables you to cancel long running queries. > > > > > > > > > > ------ > > > > Thomas White > > > > Mobile:+44 7711 922 966 > > Skype: thomaswhite > > gTalk: thomas.0007 > > Linked-In:http://www.linkedin.com/in/thomaswhite0007 > > facebook: http://www.facebook.com/thomas.0007 > > > > > > > > On 2 February 2010 16:15, Andrzej Jan Taramina <an...@ch...> > wrote: > >> > >> Thomas asked: > >> > >> >> do believe, having asynchronous mechanism for fetching data in eXist > will > >> >> > give us foundation for many?interesting new?ideas. > >> >> > I will need this functionality in about two months time and > I?really hope > >> >> > somebody from the dev team will be interested in implementing the > execution > >> >> > pipeline. > >> > >> To which Adam replied: > >> > >> > I think we will be very pushed for time to meet that deadline. The > >> > roadmap for the next two versions is already laid out - although it is > >> > subject to change. > >> > >> This feature, asynchronous execution of XQueries, is something that I'm > really interested in having as well. > >> > >> And I'm willing to implement this within the next two months (probably > sooner than that!). > >> > >> I was thinking of a new function along the lines of: > >> > >> util:eval-async( ... ) > >> > >> which would return some sort of id for the spawned xquery that could be > used to check completion status and obtain the > >> results of the execution. Same function signatures as the original > util:eval() function to start. > >> > >> Would this kind of new function meet your needs as well, Thomas? > >> > >> Thoughts from the crew on such a new function? > >> > >> > >> -- > >> Andrzej Taramina > >> Chaeron Corporation: Enterprise System Solutions > >> http://www.chaeron.com > > > > > > -- > Adam Retter > > eXist Developer > { United Kingdom } > ad...@ex... > irc://irc.freenode.net/existdb > |
|
From: Adam R. <ad...@ex...> - 2010-02-03 16:35:46
|
On 3 February 2010 15:46, Thomas White <tho...@gm...> wrote: > Adam, > > Where shell I start, may be from the fact I was not aware there are new > functions in the scheduler. Very lame :-) I dont think these are new functions, but perhaps I am not aware that someone else has augmented them? > I guess when you say "one-shot asynchronous execution" you mean > using schedule-xquery-periodic-job called with $period=0, and $repeat=1. > It seams then I have everything I need to create the batch asynch > functionality I described earlier. Yes, that is exactly the case. > BTW, is there a way to reduce the number remaining calls for > schedule-xquery-periodic-job in run time? > Use case: I need to process a dataset running periodically a XQuery function > that process a chunk of that data. I expect to have 5000 calls but the data > set is smaller this time and I need to cancel it after the 50th call. Can I > do this? If not, I hope we can then extend > schedule-xquery-periodic-job function to cancel the job if the scheduled > function returns false(). eXist maintains an Abstract idea of a Scheduler and the underlying implementation is provided by Quartz, it should be fairly easy maybe 1 - 2 hours work to abstract around the Quatz Scheduler.rescheduleJob() function and expose that functionality via the XQuery Scheduler module. > Regards, > Thomas > > ------ > > Thomas White > > Mobile:+44 7711 922 966 > Skype: thomaswhite > gTalk: thomas.0007 > Linked-In:http://www.linkedin.com/in/thomaswhite0007 > facebook: http://www.facebook.com/thomas.0007 > > > > On 3 February 2010 11:18, Adam Retter <ad...@ex...> wrote: >> >> > If we have an application where every user can fire a federated search >> > to 25 remote servers, we do need a simple and robust way to cancel >> > incompleted long running jobs, if the user makes a new federated >> > search, logs off prematurely or when the session times out. >> >> This is where I struggle to understand your vision a little bit, and >> maybe I need to revisit the original discussion. But all of what you >> want above can be achieved with the Scheduler module - the Scheduler >> module enables you to immediately Schedule an XQuery for one-shot >> asynchronous execution, it also manages the status of running Queries >> and enables you to cancel long running queries. >> >> >> > >> > >> > >> > ------ >> > >> > Thomas White >> > >> > Mobile:+44 7711 922 966 >> > Skype: thomaswhite >> > gTalk: thomas.0007 >> > Linked-In:http://www.linkedin.com/in/thomaswhite0007 >> > facebook: http://www.facebook.com/thomas.0007 >> > >> > >> > >> > On 2 February 2010 16:15, Andrzej Jan Taramina <an...@ch...> >> > wrote: >> >> >> >> Thomas asked: >> >> >> >> >> do believe, having asynchronous mechanism for fetching data in >> >> >> eXist will >> >> >> > give us foundation for many?interesting new?ideas. >> >> >> > I will need this functionality in about two months time and >> >> >> > I?really hope >> >> >> > somebody from the dev team will be interested in implementing the >> >> >> > execution >> >> >> > pipeline. >> >> >> >> To which Adam replied: >> >> >> >> > I think we will be very pushed for time to meet that deadline. The >> >> > roadmap for the next two versions is already laid out - although it >> >> > is >> >> > subject to change. >> >> >> >> This feature, asynchronous execution of XQueries, is something that I'm >> >> really interested in having as well. >> >> >> >> And I'm willing to implement this within the next two months (probably >> >> sooner than that!). >> >> >> >> I was thinking of a new function along the lines of: >> >> >> >> util:eval-async( ... ) >> >> >> >> which would return some sort of id for the spawned xquery that could be >> >> used to check completion status and obtain the >> >> results of the execution. Same function signatures as the original >> >> util:eval() function to start. >> >> >> >> Would this kind of new function meet your needs as well, Thomas? >> >> >> >> Thoughts from the crew on such a new function? >> >> >> >> >> >> -- >> >> Andrzej Taramina >> >> Chaeron Corporation: Enterprise System Solutions >> >> http://www.chaeron.com >> > >> >> >> >> -- >> Adam Retter >> >> eXist Developer >> { United Kingdom } >> ad...@ex... >> irc://irc.freenode.net/existdb > > -- Adam Retter eXist Developer { United Kingdom } ad...@ex... irc://irc.freenode.net/existdb |
|
From: Andrzej J. T. <an...@ch...> - 2010-02-03 16:10:07
|
Thomas: > BTW, is there a way to reduce the number remaining calls for > *schedule-xquery-periodic-job* in run time? > Use case: I need to process a dataset running periodically a XQuery > function that process a chunk of that data. I expect to have 5000 calls > but the data set is smaller this time and I need to cancel it after the > 50th call. Can I do this? If not, I hope we can then extend > *schedule-xquery-periodic-job* function to cancel the job if the > scheduled function returns false(). The easiest way to do that might be to just trigger the first call....and have each call chain to another call (that is, fire another scheduled job to run immediately), if there is data left to process. That way the calls will just keep being initiated until all the data is consumed and processed. -- Andrzej Taramina Chaeron Corporation: Enterprise System Solutions http://www.chaeron.com |
|
From: Adam R. <ad...@ex...> - 2010-02-03 16:36:32
|
On 3 February 2010 16:09, Andrzej Jan Taramina <an...@ch...> wrote: > Thomas: > >> BTW, is there a way to reduce the number remaining calls for >> *schedule-xquery-periodic-job* in run time? >> Use case: I need to process a dataset running periodically a XQuery >> function that process a chunk of that data. I expect to have 5000 calls >> but the data set is smaller this time and I need to cancel it after the >> 50th call. Can I do this? If not, I hope we can then extend >> *schedule-xquery-periodic-job* function to cancel the job if the >> scheduled function returns false(). > > The easiest way to do that might be to just trigger the first call....and have each call chain to another call (that is, > fire another scheduled job to run immediately), if there is data left to process. > > That way the calls will just keep being initiated until all the data is consumed and processed. Thats a much nicer approach :-) > -- > Andrzej Taramina > Chaeron Corporation: Enterprise System Solutions > http://www.chaeron.com > -- Adam Retter eXist Developer { United Kingdom } ad...@ex... irc://irc.freenode.net/existdb |
|
From: Thomas W. <tho...@gm...> - 2010-02-03 20:43:30
|
Andrzej, That will work, but it seams a little bit messy. The function has to be prepared specifically to know how to chain with re-scheduling call to itself at the end of the function. On other hand if we have schedule-xquery-periodic-job extended to pay attentions to the returned value, then any function that returns boolean can be called repeatedly until the data is consumed. Thomas ------ Thomas White Mobile:+44 7711 922 966 Skype: thomaswhite gTalk: thomas.0007 Linked-In:http://www.linkedin.com/in/thomaswhite0007 facebook: http://www.facebook.com/thomas.0007 On 3 February 2010 16:36, Adam Retter <ad...@ex...> wrote: > On 3 February 2010 16:09, Andrzej Jan Taramina <an...@ch...> > wrote: > > Thomas: > > > >> BTW, is there a way to reduce the number remaining calls for > >> *schedule-xquery-periodic-job* in run time? > >> Use case: I need to process a dataset running periodically a XQuery > >> function that process a chunk of that data. I expect to have 5000 calls > >> but the data set is smaller this time and I need to cancel it after the > >> 50th call. Can I do this? If not, I hope we can then extend > >> *schedule-xquery-periodic-job* function to cancel the job if the > >> scheduled function returns false(). > > > > The easiest way to do that might be to just trigger the first call....and > have each call chain to another call (that is, > > fire another scheduled job to run immediately), if there is data left to > process. > > > > That way the calls will just keep being initiated until all the data is > consumed and processed. > > Thats a much nicer approach :-) > > > -- > > Andrzej Taramina > > Chaeron Corporation: Enterprise System Solutions > > http://www.chaeron.com > > > > > > -- > Adam Retter > > eXist Developer > { United Kingdom } > ad...@ex... > irc://irc.freenode.net/existdb > |
|
From: Andrzej J. T. <an...@ch...> - 2010-02-03 21:21:54
|
Thomas: > That will work, but it seams a little bit messy. The function has to be > prepared specifically to know how to chain with re-scheduling call to > itself at the end of the function. True, but that is a very simple thing to do, and the XQuery would be crafted to be run using the scheduler only (though there are fancy ways to craft an XQuery that can run with a REST invocation and as a scheduled task). > On other hand if we have schedule-xquery-periodic-job extended to pay > attentions to the returned value, then any function that returns boolean > can be called repeatedly until the data is consumed. I'm gonna vote -1 on that. It's outside of the scope of what a periodic scheduled job should do and is really application specific. -- Andrzej Taramina Chaeron Corporation: Enterprise System Solutions http://www.chaeron.com |
|
From: Thomas W. <tho...@gm...> - 2010-02-04 10:28:21
|
On 3 February 2010 21:21, Andrzej Jan Taramina <an...@ch...> wrote: > > Thomas: > > > That will work, but it seams a little bit messy. The function has to be > > prepared specifically to know how to chain with re-scheduling call to > > itself at the end of the function. > > True, but that is a very simple thing to do, and the XQuery would be crafted to be run using the scheduler only (though > there are fancy ways to craft an XQuery that can run with a REST invocation and as a scheduled task). I this is an application very specific approach. I would use it only if there is no other way. > > On other hand if we have schedule-xquery-periodic-job extended to pay > > attentions to the returned value, then any function that returns boolean > > can be called repeatedly until the data is consumed. > > I'm gonna vote -1 on that. It's outside of the scope of what a periodic scheduled job should do and is really > application specific. I think the opposite way - this is generic functionality. It is an equivalent of BREAK in FOR loop. I can't think about more generic then a break functionality. One more thing. When we have an asynchronous function it is very common to have a callback function that is called at completion of the main function. If we have an option to pass a callback function when we schedule an XQuery function, this will allow us to do post processing on the results. Thomas |
|
From: Adam R. <ad...@ex...> - 2010-02-04 11:39:12
|
On 4 February 2010 10:27, Thomas White <tho...@gm...> wrote:
> On 3 February 2010 21:21, Andrzej Jan Taramina <an...@ch...> wrote:
>>
>> Thomas:
>>
>> > That will work, but it seams a little bit messy. The function has to be
>> > prepared specifically to know how to chain with re-scheduling call to
>> > itself at the end of the function.
>>
>> True, but that is a very simple thing to do, and the XQuery would be crafted to be run using the scheduler only (though
>> there are fancy ways to craft an XQuery that can run with a REST invocation and as a scheduled task).
>
> I this is an application very specific approach. I would use it only
> if there is no other way.
>
>> > On other hand if we have schedule-xquery-periodic-job extended to pay
>> > attentions to the returned value, then any function that returns boolean
>> > can be called repeatedly until the data is consumed.
>>
>> I'm gonna vote -1 on that. It's outside of the scope of what a periodic scheduled job should do and is really
>> application specific.
>
> I think the opposite way - this is generic functionality. It is an
> equivalent of BREAK in FOR loop.
> I can't think about more generic then a break functionality.
>
> One more thing. When we have an asynchronous function it is very
> common to have a callback function that is called at completion of the
> main function. If we have an option to pass a callback function when
> we schedule an XQuery function, this will allow us to do post
> processing on the results.
You can already pass functions in XQuery through the function() and
call() functions. I agree with Andrzej though. Everything that you
need is already present.
> Thomas
>
--
Adam Retter
eXist Developer
{ United Kingdom }
ad...@ex...
irc://irc.freenode.net/existdb
|
|
From: Andrzej J. T. <an...@ch...> - 2010-02-04 14:35:00
|
Thomas: > I think the opposite way - this is generic functionality. It is an > equivalent of BREAK in FOR loop. Beware of mixed metaphors. Scheduled jobs and schedulers are not the same as language design constructs. And since it's not that hard to implement what you need with existing functionality, my vote stays at -1. ;-) > One more thing. When we have an asynchronous function it is very > common to have a callback function that is called at completion of the > main function. If we have an option to pass a callback function when > we schedule an XQuery function, this will allow us to do post > processing on the results. The question with an async XQuery call is callback to where? Right now, there is no way to guarantee that the initiating XQuery is still running (XQuery being a sort-of, more or less, functional language). Having your async XQuery (initiated using the Scheduler functions) call a separate XQuery at the end to do post processing is not that difficult. -- Andrzej Taramina Chaeron Corporation: Enterprise System Solutions http://www.chaeron.com |
|
From: Thomas W. <tho...@gm...> - 2010-02-04 21:46:53
|
Andrzej and Adam, I would like to thank you both for helping me to see the simple solution that was right in front of my nose :-) Thomas ----- Original Message ----- From: "Andrzej Jan Taramina" <an...@ch...> To: "Thomas White" <tho...@gm...> Cc: "Adam Retter" <ad...@ex...>; <exi...@li...> Sent: Thursday, February 04, 2010 2:34 PM Subject: Re: Execution Pipeline, async xquery execution... > Thomas: > >> I think the opposite way - this is generic functionality. It is an >> equivalent of BREAK in FOR loop. > > Beware of mixed metaphors. Scheduled jobs and schedulers are not the same > as language design constructs. > > And since it's not that hard to implement what you need with existing > functionality, my vote stays at -1. ;-) > >> One more thing. When we have an asynchronous function it is very >> common to have a callback function that is called at completion of the >> main function. If we have an option to pass a callback function when >> we schedule an XQuery function, this will allow us to do post >> processing on the results. > > The question with an async XQuery call is callback to where? Right now, > there is no way to guarantee that the > initiating XQuery is still running (XQuery being a sort-of, more or less, > functional language). > > Having your async XQuery (initiated using the Scheduler functions) call a > separate XQuery at the end to do post > processing is not that difficult. > > -- > Andrzej Taramina > Chaeron Corporation: Enterprise System Solutions > http://www.chaeron.com > |
|
From: Andrzej J. T. <an...@ch...> - 2010-02-04 21:53:28
|
Thomas: > I would like to thank you both for helping me to see the simple solution > that was right in front of my nose :-) No worries! -- Andrzej Taramina Chaeron Corporation: Enterprise System Solutions http://www.chaeron.com |
|
From: Adam R. <ad...@ex...> - 2010-02-03 11:17:12
|
Perhaps it is also interesting to consider the development of a pragma -
(# exist:thread #){
(: asynchronous xquery code :)
}
On 2 February 2010 16:15, Andrzej Jan Taramina <an...@ch...> wrote:
> Thomas asked:
>
>>> do believe, having asynchronous mechanism for fetching data in eXist will
>>> > give us foundation for many?interesting new?ideas.
>>> > I will need this functionality in about two months time and I?really hope
>>> > somebody from the dev team will be interested in implementing the execution
>>> > pipeline.
>
> To which Adam replied:
>
>> I think we will be very pushed for time to meet that deadline. The
>> roadmap for the next two versions is already laid out - although it is
>> subject to change.
>
> This feature, asynchronous execution of XQueries, is something that I'm really interested in having as well.
>
> And I'm willing to implement this within the next two months (probably sooner than that!).
>
> I was thinking of a new function along the lines of:
>
> util:eval-async( ... )
>
> which would return some sort of id for the spawned xquery that could be used to check completion status and obtain the
> results of the execution. Same function signatures as the original util:eval() function to start.
>
> Would this kind of new function meet your needs as well, Thomas?
>
> Thoughts from the crew on such a new function?
>
>
> --
> Andrzej Taramina
> Chaeron Corporation: Enterprise System Solutions
> http://www.chaeron.com
>
--
Adam Retter
eXist Developer
{ United Kingdom }
ad...@ex...
irc://irc.freenode.net/existdb
|
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X