xsltforms-support

[Xsltforms-support] Evaluating long calculations asynchronously in background

[Mats E. <mat...@li...>]

Hi,
I am looking at options for performing time-consuming operations, like...
    <xf:setvalue ref="mynode" value="digest(transform(instance('my-very-large-instance-document')),'SHA-1')"/>
...in my form as background operations that don't cause freezing of the form's UI. 
Obviously, if my form had access to a web-server, such calculations could be posted to the server using an xf:submission, however, in my case, the form should work completely offline.
In HTML 5 there is the Web Worker object, and it should be possible to leverage this to perform the calculation in a background thread client-side.
I guess it is possible to implement the long running calculation in a javascript function that may be called via xf:load and where the result is returned via call to XForms DOM API (model.getInstanceDocument(), model.recalculate()...).
However, it would be nice if the operation could be kept more "XFormy" and as declarative as in the "non-async" example above.
I'm currently considering two possible options:
1) XForms extension attribute, like so:
    <xf:setvalue ref="mynode" value="..." @ext:async="true"/>
2) Implementing the operation as an xf:submission, but proxying the underlying XMLHttpRequest object to intercept selected submission URLs and handle them client-side:
    <xf:submission resource="http://client/myfunction?param1=..." serialization="...".../>
    and
    XMLHttpRequest.registerHandler("http://client/myfunction", myhandler)

Both options seem to have their unique merits and possible usages. While the second option is probably quite straight-forward to implement, the second option may require some more work, like:
1) Porting XSLTForms XPath implementation, as well as some other objects (XSLTTransform), to work in a Web Worker context.2) Finding a way to efficiently transfer relevant data (XForms instances) to the Worker context2) Making modifications to the relevant action implementations in XSLTForms to handle asynchronous execution of the action
Seems doable to me, but would be interested to hear if someone has experience to tell whether that'a crazy idea or not. Obviously, would be cool if such features could become native features of XSLTForms!
Welcoming your feedback!
Mats
 		 	   		  

Re: [Xsltforms-support] Evaluating long calculations asynchronously in background

[Alain C. <ala...@ag...>]

Hello Mats,
> In HTML 5 there is the Web Worker object, and it should be possible to 
> leverage this to perform the calculation in a background thread 
> client-side.
XSLTForms is not yet using Web Worker but Google Chrome is already 
complaining about XMLHttpRequest being called in main thread... ;-)
>
> I guess it is possible to implement the long running calculation in a 
> javascript function that may be called via xf:load and where the 
> result is returned via call to XForms DOM API 
> (model.getInstanceDocument(), model.recalculate()...).
>
> However, it would be nice if the operation could be kept more "XFormy" 
> and as declarative as in the "non-async" example above.
>
> I'm currently considering two possible options:
>
> 1) XForms extension attribute, like so:
>
> <xf:setvalue ref="mynode" value="..." @ext:async="true"/>
>
> 2) Implementing the operation as an xf:submission, but proxying the 
> underlying XMLHttpRequest object to intercept selected submission URLs 
> and handle them client-side:
>
>     <xf:submission resource="http://client/myfunction?param1=..." 
> serialization="...".../>
>
>     and
>
> XMLHttpRequest.registerHandler("http://client/myfunction", myhandler)
>
XSLTForms supports xf:setnode as an extension. So, with both xf:setvalue 
and xf:setnode, at first glance, I prefer a 
@mode="synchronous/asynchronous" implementation (as in xf:submission).
>
> Both options seem to have their unique merits and possible usages. 
> While the second option is probably quite straight-forward to 
> implement, the second option may require some more work, like:
>
> 1) Porting XSLTForms XPath implementation, as well as some other 
> objects (XSLTTransform), to work in a Web Worker context.
I have not been working with Web Workers yet but I read that there are 
limitations like these. XSLTForms already has its own XPath engine but 
is currently using native XML engines (an independent XML engine, fully 
written in Javascript, is already in good progress!). Why do you think 
that XSLT would also be required or is just it that XSLT would be nice 
to have available too?
> 2) Finding a way to efficiently transfer relevant data (XForms 
> instances) to the Worker context
> 2) Making modifications to the relevant action implementations in 
> XSLTForms to handle asynchronous execution of the action
>
> Seems doable to me, but would be interested to hear if someone has 
> experience to tell whether that'a crazy idea or not. Obviously, would 
> be cool if such features could become native features of XSLTForms!
I will be happy to add such features in XSLTForms even if only recent 
browsers will support them!

What do you think?

--Alain

Re: [Xsltforms-support] Evaluating long calculations asynchronously in background

[Mats E. <mat...@li...>]

Hi Alain,
Thanks for your feedback!
> XSLTForms supports xf:setnode as an extension. So, with both
    xf:setvalue and xf:setnode, at first glance, I prefer a
    @mode="synchronous/asynchronous" implementation (as in
    xf:submission).
One of the advantages I saw with the xf:submission + proxied XMLHttpRequest option is the submission-done notification event, which allows chaining subsequent actions. I now realize however, that a similar notification event is triggered by the xf:insert action (xforms-insert), so if the @asynchronous attribute could be applied on this action, the extension attribute approach would probably be the more interesting option. Furthermore, it looks like both xf:setvalue and xf:setnode could be replaced by an xf:insert action if the XPath 3.0 functions fn:parse-xml() and fn:parse-xml-fragment() were available in XLSTForms (for use in the xf:insert/@origin attribute), in cases where the notification event is important. This seems doable as well...
> I have not been working with Web Workers yet but I read that there
    are limitations like these. XSLTForms already has its own XPath
    engine but is currently using native XML engines (an independent XML
    engine, fully written in Javascript, is already in good progress!).
    Why do you think that XSLT would also be required or is just it that
    XSLT would be nice to have available too?
Concerning XSLTProcessor, large XSLT transforms can be long running so it would be nice if such could be run in a background thread. Furthermore, the transform() xpath function is already part of the XSLTForms core function library, and I'm thinking this entire library should ideally be available also in the web worker context. I see there are some javascript implementations of XSLTProcessors on the web (not sure if Saxon CE is possible to use -- I'm trying to clarify). 
By the way, which XML DOM level is required by XSLTForms' XPath implementation for navigating on XForms instance documents, do you know? So far it looks like the browsers' native XML DOM Parser is not available in web worker context...
      
/ Mats
     		 	   		  

Re: [Xsltforms-support] Evaluating long calculations asynchronously in background

[Alain C. <ala...@ag...>]

Hi Mats,

The more I think about using a Web Worker for XSLTForms, the more it 
sounds obvious that the right solution should be to dedicate a Web 
Worker for storing/manipulating instances (XPath evaluations and 
submissions): only node lists and values are to be present in main 
thread actually!

First, this means that source code has to be adapted in a lot of 
different locations.

Second, my Javascript XML DOM implementation is ready for integration 
but my new XPath engine is not yet operational except for very simple 
expressions...

What do you think of this different approach?

--Alain

Le 31/08/2015 07:25, Mats Eklund a écrit :
> Hi Alain,
>
> Thanks for your feedback!
>
>     >XSLTForms supports xf:setnode as an extension. So, with both
>     xf:setvalue and xf:setnode, at first glance, I prefer a
>     @mode="synchronous/asynchronous" implementation (as in xf:submission).
>
> One of the advantages I saw with the xf:submission + proxied 
> XMLHttpRequest option is the submission-done notification event, which 
> allows chaining subsequent actions. I now realize however, that a 
> similar notification event is triggered by the xf:insert action 
> (xforms-insert), so if the @asynchronous attribute could be applied on 
> this action, the extension attribute approach would probably be the 
> more interesting option. Furthermore, it looks like both xf:setvalue 
> and xf:setnode could be replaced by an xf:insert action if the XPath 
> 3.0 functions fn:parse-xml() and fn:parse-xml-fragment() were 
> available in XLSTForms (for use in the xf:insert/@origin attribute), 
> in cases where the notification event is important. This seems doable 
> as well...
>
>     > I have not been working with Web Workers yet but I read that
>     there are limitations like these. XSLTForms already has its own
>     XPath engine but is currently using native XML engines (an
>     independent XML engine, fully written in Javascript, is already in
>     good progress!). Why do you think that XSLT would also be required
>     or is just it that XSLT would be nice to have available too?
>
> Concerning XSLTProcessor, large XSLT transforms can be long running so 
> it would be nice if such could be run in a background thread. 
> Furthermore, the transform() xpath function is already part of the 
> XSLTForms core function library, and I'm thinking this entire library 
> should ideally be available also in the web worker context. I see 
> there are some javascript implementations of XSLTProcessors on the web 
> (not sure if Saxon CE is possible to use -- I'm trying to clarify).
>
> By the way, which XML DOM level is required by XSLTForms' XPath 
> implementation for navigating on XForms instance documents, do you 
> know? So far it looks like the browsers' native XML DOM Parser is not 
> available in web worker context...
>
> / Mats
>
>
> ------------------------------------------------------------------------------
>
>
> _______________________________________________
> Xsltforms-support mailing list
> Xsl...@li...
> https://lists.sourceforge.net/lists/listinfo/xsltforms-support

Re: [Xsltforms-support] Evaluating long calculations asynchronously in background

[Mats E. <mat...@li...>]

Hi Alain,
Sounds like an interesting idea. Probably not trivial though! 
For the problem I have (long running calculations) I think it would still be necessary to be able to spawn off a dedicated thread (separate from the main execution thread) for performing the long running calculation (in order that the form is still responsive to other events), and a way to indicate that an action should be executed in a non-blocking manner is required I believe. While migrating all xpath evaluations to a (single) separate web worker might not solve the problem, the implementation of a feature for executing long running calculations in a separate worker thread would probably be simplified if the framework already dealt with web workers in some way or the other...
In any case, I am progressing with the implementation of the @mode=async attribute (working off your v595), which I hope shall work on xf:setvalue, xf:insert and maybe xf:setnode. 
The approach I am taking involves serializing the XsltForms_xpath and XsltForms_exprContext objects (including relevant references), passing them to a new Worker thread, deserializing and evaluating the expressions there, then serializing the result, returning it to the caller thread and deserializing the result there to allow completion of the setvalue/insert/setnode action. All these steps happen inside a new method on the XsltForms_xpath object called xpath_evaluate_async() which must be called by the action instead of the normal xpath_evaluate(). The new method returns a Promise. (I am currently using a 3rd party serialization library... there is another library out there called vkThread which could also be interesting to look at). For time being, I am only considering evaluating the xpath expressions in the @value, @origin and @inner/@outer attributes in this manner. Binding expressions etc will be evaluated in the main thread as they are currently. 
I realize that most likely my implementation will have some limitations (e.g. some xpath functions not supported since they have dependencies that might not be transferable, not all return types supported etc.) -- some which might be overcome with additional effort -- however I think most important use cases can be supported still with such limitations (e.g. calculating digests etc)...
I am interested to learn what you have in planning for a new XPath engine!? If the ability to evaluate expressions in web worker context is taken into account in the design, that would be great of course!
Welcome your feedback!

Mats


On Mon, 31 Aug 2015 21:18:53 +0200, Alain Couthures wrote:


  
    
  
  
    Hi Mats,

      The more I think about using a Web Worker for XSLTForms, the more
      it sounds obvious that the right solution should be to dedicate a
      Web Worker for storing/manipulating instances (XPath evaluations
      and submissions): only node lists and values are to be present in
      main thread actually!

      First, this means that source code has to be adapted in a lot of
      different locations.

      Second, my Javascript XML DOM implementation is ready for
      integration but my new XPath engine is not yet operational except
      for very simple expressions...

      What do you think of this different approach?

      --Alain

      Le 31/08/2015 07:25, Mats Eklund a écrit :Hi Alain,
        
Thanks for your feedback!
> XSLTForms supports
                    xf:setnode as an extension. So, with both
                    xf:setvalue and xf:setnode, at first glance, I
                    prefer a @mode="synchronous/asynchronous"
                    implementation (as in xf:submission).
One of the advantages I saw
              with the xf:submission + proxied XMLHttpRequest option is
              the submission-done notification event, which allows
              chaining subsequent actions. I now realize however, that a
              similar notification event is triggered by the xf:insert
              action (xforms-insert), so if the @asynchronous attribute
              could be applied on this action, the extension attribute
              approach would probably be the more interesting option.
              Furthermore, it looks like both xf:setvalue and xf:setnode
              could be replaced by an xf:insert action if the XPath 3.0
              functions fn:parse-xml() and fn:parse-xml-fragment() were
              available in XLSTForms (for use in the xf:insert/@origin
              attribute), in cases where the notification event is
              important. This seems doable as well...
> I have not been working with Web Workers yet but I
              read that there are limitations like these. XSLTForms
              already has its own XPath engine but is currently using
              native XML engines (an independent XML engine, fully
              written in Javascript, is already in good progress!). Why
              do you think that XSLT would also be required or is just
              it that XSLT would be nice to have available too?
Concerning XSLTProcessor, large
          XSLT transforms can be long running so it would be nice if
          such could be run in a background thread. Furthermore, the
          transform() xpath function is already part of the XSLTForms
          core function library, and I'm thinking this entire library
          should ideally be available also in the web worker context. I see there are some javascript
          implementations of XSLTProcessors on the web (not sure if
          Saxon CE is possible to use -- I'm trying to clarify). 
By the way, which XML DOM level is required by XSLTForms'
          XPath implementation for navigating on XForms instance
          documents, do you know? So far it looks like the browsers'
          native XML DOM Parser is not available in web worker
          context...
          

                  / Mats 

------------------------------------------------------------------------------

_______________________________________________Xsltforms-support mailing lis...@li...://lists.sourceforge.net/lists/listinfo/xsltforms-support
    
    
 		 	   		  

Re: [Xsltforms-support] Evaluating long calculations asynchronously in background

[Mats E. <mat...@li...>]

Btw, concerning how to pass the compiled xpath object to the web worker, I was considering the option of rebuilding the xpath object from its expression from within the worker, however, XsltForms' xpath parser is available only as an XSLT tranform, so I figured the serialization option was probably the path of less resistance...
/ Mats


On Tue, 1 Sep 2015 15:04:39 +0000, Mats wrote:

Hi Alain,
Sounds like an interesting idea. Probably not trivial though! 
For the problem I have (long running calculations) I think it would still be necessary to be able to spawn off a dedicated thread (separate from the main execution thread) for performing the long running calculation (in order that the form is still responsive to other events), and a way to indicate that an action should be executed in a non-blocking manner is required I believe. While migrating all xpath evaluations to a (single) separate web worker might not solve the problem, the implementation of a feature for executing long running calculations in a separate worker thread would probably be simplified if the framework already dealt with web workers in some way or the other...
In any case, I am progressing with the implementation of the @mode=async attribute (working off your v595), which I hope shall work on xf:setvalue, xf:insert and maybe xf:setnode. 
The approach I am taking involves serializing the XsltForms_xpath and XsltForms_exprContext objects (including relevant references), passing them to a new Worker thread, deserializing and evaluating the expressions there, then serializing the result, returning it to the caller thread and deserializing the result there to allow completion of the setvalue/insert/setnode action. All these steps happen inside a new method on the XsltForms_xpath object called xpath_evaluate_async() which must be called by the action instead of the normal xpath_evaluate(). The new method returns a Promise. (I am currently using a 3rd party serialization library... there is another library out there called vkThread which could also be interesting to look at). For time being, I am only considering evaluating the xpath expressions in the @value, @origin and @inner/@outer attributes in this manner. Binding expressions etc will be evaluated in the main thread as they are currently. 
I realize that most likely my implementation will have some limitations (e.g. some xpath functions not supported since they have dependencies that might not be transferable, not all return types supported etc.) -- some which might be overcome with additional effort -- however I think most important use cases can be supported still with such limitations (e.g. calculating digests etc)...
I am interested to learn what you have in planning for a new XPath engine!? If the ability to evaluate expressions in web worker context is taken into account in the design, that would be great of course!
Welcome your feedback!
Mats


On Mon, 31 Aug 2015 21:18:53 +0200, Alain Couthures wrote:
Hi Mats,

      The more I think about using a Web Worker for XSLTForms, the more
      it sounds obvious that the right solution should be to dedicate a
      Web Worker for storing/manipulating instances (XPath evaluations
      and submissions): only node lists and values are to be present in
      main thread actually!

      First, this means that source code has to be adapted in a lot of
      different locations.

      Second, my Javascript XML DOM implementation is ready for
      integration but my new XPath engine is not yet operational except
      for very simple expressions...

      What do you think of this different approach?

      --Alain

      Le 31/08/2015 07:25, Mats Eklund a écrit :Hi Alain,
        
Thanks for your feedback!
> XSLTForms supports
                    xf:setnode as an extension. So, with both
                    xf:setvalue and xf:setnode, at first glance, I
                    prefer a @mode="synchronous/asynchronous"
                    implementation (as in xf:submission).
One of the advantages I saw
              with the xf:submission + proxied XMLHttpRequest option is
              the submission-done notification event, which allows
              chaining subsequent actions. I now realize however, that a
              similar notification event is triggered by the xf:insert
              action (xforms-insert), so if the @asynchronous attribute
              could be applied on this action, the extension attribute
              approach would probably be the more interesting option.
              Furthermore, it looks like both xf:setvalue and xf:setnode
              could be replaced by an xf:insert action if the XPath 3.0
              functions fn:parse-xml() and fn:parse-xml-fragment() were
              available in XLSTForms (for use in the xf:insert/@origin
              attribute), in cases where the notification event is
              important. This seems doable as well...
> I have not been working with Web Workers yet but I
              read that there are limitations like these. XSLTForms
              already has its own XPath engine but is currently using
              native XML engines (an independent XML engine, fully
              written in Javascript, is already in good progress!). Why
              do you think that XSLT would also be required or is just
              it that XSLT would be nice to have available too?
Concerning XSLTProcessor, large
          XSLT transforms can be long running so it would be nice if
          such could be run in a background thread. Furthermore, the
          transform() xpath function is already part of the XSLTForms
          core function library, and I'm thinking this entire library
          should ideally be available also in the web worker context. I see there are some javascript
          implementations of XSLTProcessors on the web (not sure if
          Saxon CE is possible to use -- I'm trying to clarify). 
By the way, which XML DOM level is required by XSLTForms'
          XPath implementation for navigating on XForms instance
          documents, do you know? So far it looks like the browsers'
          native XML DOM Parser is not available in web worker
          context...
          

                  / Mats 

------------------------------------------------------------------------------

_______________________________________________Xsltforms-support mailing lis...@li...://lists.sourceforge.net/lists/listinfo/xsltforms-support