betterform-developer — developer discussions

Re: [Betterform-developer] UI optimizations

[Ronald v. K. <Rva...@de...>]

> 
> ________________________________________
> Van: Joern Turner [joe...@be...]
> Verzonden: vrijdag 15 februari 2013 18:57
> Aan: bet...@li...
> Onderwerp: Re: [Betterform-developer] UI optimizations
> 
> Am 15.02.13 12:37, schrieb Ronald van Kuijk:
> > Guys,
> >
> > First of all, sorry for the huge amount of text, but it unfortunately is
> > necessary. The corresponding form and additional files can not be send
> > to the list, so will send them directly.
> >
> > We were having some performance problems with a certain form we use. It
> > basically comes down to a 'create order' solution, where you can select
> > articles in a select1 in a dialog, and add them to the order. The
> > article line contains some calculations but nothing realy fancy (imo).
> >
> > The performance issues were mainly in IE8 and 9, but FF and Chrome were
> > also not very good. The first bottleneck was using the full article
> > information (500 articles, 50MB as a dom!) from a separate instance
> > backing the select1. This instance was refreshed etc each time something
> > changed in the order instances since it was in the same model. Putting
> > it in a separtate model helps (thanks to the instance-of-model
> > function!!!). We might even look into cancelling refresh events etc on
> > this model to increase performance even more. In addition we created a
> > more minimal list of article information so memory usage was waaay
> > better. Not sure why untouched/unchanged instances were fully refreshed
> > etc, but never the less, this 'workaround' is acceptable...
> It's a common issue with XForms that big itemsets are an issue mostly
> due to the fact that they are completely loaded as DOM. Certain
> implementations have workarounds for that. E.g. i think the IBM
> implementation uses a special kind of instance that is not handled as a
> normal instance.

As does Orbeon afaik...

> In the past we also had similar problems and used some hacks to get
> around it. One approach is to serve the data that normally would be
> handled by the itemset directly to the ui control with some ajax
> request. So the respective select wouldn't use the itemset but would
> request the data directly from the browser. Of course this requires a
> custom ui control. This means that these data are not handled in XForms
> at all but processed by the side.
> 
> However this approach is neither very elegant nor generic.
> 

Agreed

> > - In the inserts the value and label are empty, separate item-changed
> > events happen after that, 1 for the label AND one for the value, not in
> > one operation! I know that in the xforms standard they are separate, but
> > I hope this can (and I volunteer to try) be optimized.
> >
> > Any thoughts?
> Seems we are having two aspects here: first the memory consumption for a
> large itemset list and second the amount of messages exchanged to handle
> the labels, values.
> 
> We are very aware of the problem and have already thought about
> improvements. First it would be good to avoid the overhead of parsing
> the data for an itemset into an instance as it's normally just readonly
> data that you just put into an instance for the purpose of serving them
> to the control. There's not much advantage of having them as an
> instance. We are planning to extend the itemset element to also directly
> accept an URI where to load the data without parsing those into xml. The
> URI could for instance just be passed on to the browser that then in
> turn would use some ajax to fetch the items. This also allows for the
> usage of JSON as a more effective format for transfer.
> 
I don't mind having the data in xml form on the server, nor as an instance... It is the amount of data send from the server to the client and the processing required on the client. Sending the  'html' directly would even be more performant. Using JSON as a client-server communication instead of DWR would also help a lot I think...

> Another thing would be to support readonly instances (as others already
> do). This can be done by adding a bind to the root node of the instance
> that set the readonly mip to true(). As these data cannot change there
> would be room for event optimization as it's not necessary to refresh
> such instances.

We have a kind of patch for this to... it currently 'requires' a bf:dataOnly attribute on the instance (attribute can ofcourse be renamed... ) but using the bind is another option as well... 

> 
> This is just a quick guess and there's surely more that can be done.
>

Re: [Betterform-developer] UI optimizations

[Joern T. <joe...@be...>]

Am 15.02.13 12:37, schrieb Ronald van Kuijk:
> Guys,
>
> First of all, sorry for the huge amount of text, but it unfortunately is
> necessary. The corresponding form and additional files can not be send
> to the list, so will send them directly.
>
> We were having some performance problems with a certain form we use. It
> basically comes down to a 'create order' solution, where you can select
> articles in a select1 in a dialog, and add them to the order. The
> article line contains some calculations but nothing realy fancy (imo).
>
> The performance issues were mainly in IE8 and 9, but FF and Chrome were
> also not very good. The first bottleneck was using the full article
> information (500 articles, 50MB as a dom!) from a separate instance
> backing the select1. This instance was refreshed etc each time something
> changed in the order instances since it was in the same model. Putting
> it in a separtate model helps (thanks to the instance-of-model
> function!!!). We might even look into cancelling refresh events etc on
> this model to increase performance even more. In addition we created a
> more minimal list of article information so memory usage was waaay
> better. Not sure why untouched/unchanged instances were fully refreshed
> etc, but never the less, this 'workaround' is acceptable...
It's a common issue with XForms that big itemsets are an issue mostly 
due to the fact that they are completely loaded as DOM. Certain 
implementations have workarounds for that. E.g. i think the IBM 
implementation uses a special kind of instance that is not handled as a 
normal instance.

In the past we also had similar problems and used some hacks to get 
around it. One approach is to serve the data that normally would be 
handled by the itemset directly to the ui control with some ajax 
request. So the respective select wouldn't use the itemset but would 
request the data directly from the browser. Of course this requires a 
custom ui control. This means that these data are not handled in XForms 
at all but processed by the side.

However this approach is neither very elegant nor generic.

>
> But even then, the performance of initially populating the select1 was
> bad... 5-10 seconds. I looked into the request-response of the opening
> of the dialog, and noticed that for adding even a mere 150 articles into
> the select1, (20 chars  for the label, 10 for the value, so 3k of data)
> almost 250kB was transfered... that is 1.7k per article or a 5500%
> overhead!!! Wow.... Ok, IE8 is not fast in dom manipulation (FF and
> Chrome were somewhat acceptable.
>
> This was all with 4.1. Since we want to start migrating to 5.0, I
> thought I'd give that a try. The 'dwr' response here was kind of the
> same size (>3000 lines!!!) , so that was not an advantage (I think using
> DWR is not an advantage here)
>
> Things I witnessed after some more debugging
> -  a huge performance issue with dom manipulation in Select1ComboBox.
> The 'getNthSibblingOption' loops waaaay to much over dom elements if an
> itemset is populated at once... siblingNode = siblingNode.nextSibling is
> called n(n+1)/2 times!!! or 11325 times.... I created a patch
> <https://github.com/betterFORM/betterFORM/pull/11> that reduces this to
> 150.
wow - that sounds like an impressive improvement. Of course we're very 
interested in your patches.

> - In the inserts the value and label are empty, separate item-changed
> events happen after that, 1 for the label AND one for the value, not in
> one operation! I know that in the xforms standard they are separate, but
> I hope this can (and I volunteer to try) be optimized.
>
> Any thoughts?
Seems we are having two aspects here: first the memory consumption for a 
large itemset list and second the amount of messages exchanged to handle 
the labels, values.

We are very aware of the problem and have already thought about 
improvements. First it would be good to avoid the overhead of parsing 
the data for an itemset into an instance as it's normally just readonly 
data that you just put into an instance for the purpose of serving them 
to the control. There's not much advantage of having them as an 
instance. We are planning to extend the itemset element to also directly 
accept an URI where to load the data without parsing those into xml. The 
URI could for instance just be passed on to the browser that then in 
turn would use some ajax to fetch the items. This also allows for the 
usage of JSON as a more effective format for transfer.

Another thing would be to support readonly instances (as others already 
do). This can be done by adding a bind to the root node of the instance 
that set the readonly mip to true(). As these data cannot change there 
would be room for event optimization as it's not necessary to refresh 
such instances.

This is just a quick guess and there's surely more that can be done.

>
> Ronald
>
>
>
>
>
>
> ------------------------------------------------------------------------------
> Free Next-Gen Firewall Hardware Offer
> Buy your Sophos next-gen firewall before the end March 2013
> and get the hardware for free! Learn more.
> http://p.sf.net/sfu/sophos-d2d-feb
>
>
>
> _______________________________________________
> Betterform-developer mailing list
> Bet...@li...
> https://lists.sourceforge.net/lists/listinfo/betterform-developer
>

[Betterform-developer] UI optimizations

[Ronald v. K. <Rva...@de...>]

Guys,

First of all, sorry for the huge amount of text, but it unfortunately is necessary. The corresponding form and additional files can not be send to the list, so will send them directly.

We were having some performance problems with a certain form we use. It basically comes down to a 'create order' solution, where you can select articles in a select1 in a dialog, and add them to the order. The article line contains some calculations but nothing realy fancy (imo).

The performance issues were mainly in IE8 and 9, but FF and Chrome were also not very good. The first bottleneck was using the full article information (500 articles, 50MB as a dom!) from a separate instance backing the select1. This instance was refreshed etc each time something changed in the order instances since it was in the same model. Putting it in a separtate model helps (thanks to the instance-of-model function!!!). We might even look into cancelling refresh events etc on this model to increase performance even more. In addition we created a more minimal list of article information so memory usage was waaay better. Not sure why untouched/unchanged instances were fully refreshed etc, but never the less, this 'workaround' is acceptable...

But even then, the performance of initially populating the select1 was bad... 5-10 seconds. I looked into the request-response of the opening of the dialog, and noticed that for adding even a mere 150 articles into the select1, (20 chars  for the label, 10 for the value, so 3k of data) almost 250kB was transfered... that is 1.7k per article or a 5500% overhead!!! Wow.... Ok, IE8 is not fast in dom manipulation (FF and Chrome were somewhat acceptable.

This was all with 4.1. Since we want to start migrating to 5.0, I thought I'd give that a try. The 'dwr' response here was kind of the same size (>3000 lines!!!) , so that was not an advantage (I think using DWR is not an advantage here)

Things I witnessed after some more debugging
-  a huge performance issue with dom manipulation in Select1ComboBox. The 'getNthSibblingOption' loops waaaay to much over dom elements if an itemset is populated at once... siblingNode = siblingNode.nextSibling is called n(n+1)/2 times!!! or 11325 times.... I created a patch <https://github.com/betterFORM/betterFORM/pull/11> that reduces this to 150.
- In the inserts the value and label are empty, separate item-changed events happen after that, 1 for the label AND one for the value, not in one operation! I know that in the xforms standard they are separate, but I hope this can (and I volunteer to try) be optimized.

Any thoughts?

Ronald

Re: [Betterform-developer] Compressed AND uncompressed version of js?

[Ronald v. K. <Rva...@de...>]

> And still, this would not work in a preprod environment, buidling a
> new version just to do debugging is not an option. Therefor the
> question about 'runtime' or config time changing of what kind of
> resources are used.
No, we also do not rebuild.

It's not so much the rebuilding, but getting this changed version through all internal procedures, deployed and then afterwards undeployed and replaced with a normal version. Way to much trouble.... And a collegue working on an application that embeds BF can almost do no debugging at all if he/she does not want to checkout the BF source etc... So it is more (and I understand that this might not be an issue for you guys)

>>
>> Keep in mind that I and one of my collegues developon the ui
>> components and and therefor need to the debug the components. So
>> using the optimized version from BF is not an option since changes we
>> make end up in BF a little later (no problem, just a fact)
>>
>> What I am thinking about is a kind of switch (e.g. a debug-all) that
>> besides enabeling the BF debug pane, also starts using the
>> non-optimized versions of the script.
> Well, that would come to the cost of having 2 versions deployed right?

Well, yes, but with an added ant target that either just does the optimized stuff or a combination

> But of course that should be possible to do. I assume that you would
> modify the ResourceServlet to serve from a different location when your
> switch applies.

Yep, that is the idea... maybe in combination with the debug param already in the xslt 


Ronald

Re: [Betterform-developer] Compressed AND uncompressed version of js?

[Joern T. <joe...@be...>]

Am 17.09.12 16:38, schrieb Ronald van Kuijk:
> :-)
>
> We use betterform embedded in a different application, so calling ant
> deploy and ant deploy-exploded does not work (why do you think I want
> to use maven more ;-))
>
> And still, this would not work in a preprod environment, buidling a
> new version just to do debugging is not an option. Therefor the
> question about 'runtime' or config time changing of what kind of
> resources are used.
No, we also do not rebuild.
>
> Keep in mind that I and one of my collegues developon the ui
> components and and therefor need to the debug the components. So
> using the optimized version from BF is not an option since changes we
> make end up in BF a little later (no problem, just a fact)
>
> What I am thinking about is a kind of switch (e.g. a debug-all) that
> besides enabeling the BF debug pane, also starts using the
> non-optimized versions of the script.
Well, that would come to the cost of having 2 versions deployed right? 
But of course that should be possible to do. I assume that you would 
modify the ResourceServlet to serve from a different location when your 
switch applies.


>
> Cheers,
>
> Ronald ________________________________________ From: Joern Turner
> [joe...@be...] Sent: Monday, September 17, 2012 4:28
> PM To: Ronald van Kuijk Cc:
> bet...@li... Subject: Re:
> [Betterform-developer] Compressed AND uncompressed version of js?
>
> Am 17.09.12 15:54, schrieb Ronald van Kuijk:
>> Hi,
>>
>> Is it in one way or another possible to have both the compressed
>> and uncompressed versions of the javascript in both a exploded
>> deploy and the single jar deploy? And preferably switch by e.g.
>> setting the debug flag in the config?
>
> No, not at the moment. That's because the resources are loaded from
> the classpath (always META-INF/resources regardless of exploded or
> jar-file mode). The advantage is that we always use the same path and
> do not need to switch patches in xslt etc. That's why we went this
> road. For us the switching is as simple as calling deploy-exploded or
> simply deploy.
>
>>
>> That way it is fairly easy to switch from develop to production
>> mode. It is e.g. sometimes handy if we can debug on 'preprod'
>> systems without having to create a special build
> Not sure if i get it. We have the ready-built version of Dojo around
> in the repo - so it's a matter of deployment (copying that) and you
> don't need to rebuild that all the time.
>
>
>>
>> Cheers,
>>
>> Ronald
>>
>>
>> ------------------------------------------------------------------------------
>>
>>
Live Security Virtual Conference
>> Exclusive live event will cover all the ways today's security and
>> threat landscape has changed and how IT managers can respond.
>> Discussions will include endpoint security, mobile security and the
>> latest in malware threats.
>> http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>
>>
>>
>> _______________________________________________
>> Betterform-developer mailing list
>> Bet...@li...
>> https://lists.sourceforge.net/lists/listinfo/betterform-developer
>>
>
>
>

Re: [Betterform-developer] Compressed AND uncompressed version of js?

[Ronald v. K. <Rva...@de...>]

:-)

We use betterform embedded in a different application, so calling ant deploy and ant deploy-exploded does not work (why do you think I want to use maven more ;-))

And still, this would not work in a preprod environment, buidling a new version just to do debugging is not an option. Therefor the question about 'runtime' or config time changing of what kind of resources are used.

Keep in mind that I and one of my collegues developon the ui components and and therefor need to the debug the components. So using the optimized version from BF is not an option since changes we make end up in BF a little later (no problem, just a fact)

What I am thinking about is a kind of switch (e.g. a debug-all) that besides enabeling the BF debug pane, also starts using the non-optimized versions of the script.

Cheers,

Ronald
________________________________________
From: Joern Turner [joe...@be...]
Sent: Monday, September 17, 2012 4:28 PM
To: Ronald van Kuijk
Cc: bet...@li...
Subject: Re: [Betterform-developer] Compressed AND uncompressed version of js?

Am 17.09.12 15:54, schrieb Ronald van Kuijk:
> Hi,
>
> Is it in one way or another possible to have both the compressed and
> uncompressed versions of the javascript in both a exploded deploy and
> the single jar deploy? And preferably switch by e.g. setting the debug
> flag in the config?

No, not at the moment. That's because the resources are loaded from the
classpath (always META-INF/resources regardless of exploded or jar-file
mode). The advantage is that we always use the same path and do not need
to switch patches in xslt etc. That's why we went this road. For us the
switching is as simple as calling deploy-exploded or simply deploy.

>
> That way it is fairly easy to switch from develop to production mode. It
> is e.g. sometimes handy if we can debug on 'preprod' systems without
> having to create a special build
Not sure if i get it. We have the ready-built version of Dojo around in
the repo - so it's a matter of deployment (copying that) and you don't
need to rebuild that all the time.


>
> Cheers,
>
> Ronald
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>
>
>
> _______________________________________________
> Betterform-developer mailing list
> Bet...@li...
> https://lists.sourceforge.net/lists/listinfo/betterform-developer
>

Re: [Betterform-developer] Compressed AND uncompressed version of js?

[Joern T. <joe...@be...>]

Am 17.09.12 15:54, schrieb Ronald van Kuijk:
> Hi,
>
> Is it in one way or another possible to have both the compressed and
> uncompressed versions of the javascript in both a exploded deploy and
> the single jar deploy? And preferably switch by e.g. setting the debug
> flag in the config?

No, not at the moment. That's because the resources are loaded from the 
classpath (always META-INF/resources regardless of exploded or jar-file 
mode). The advantage is that we always use the same path and do not need 
to switch patches in xslt etc. That's why we went this road. For us the 
switching is as simple as calling deploy-exploded or simply deploy.

>
> That way it is fairly easy to switch from develop to production mode. It
> is e.g. sometimes handy if we can debug on 'preprod' systems without
> having to create a special build
Not sure if i get it. We have the ready-built version of Dojo around in 
the repo - so it's a matter of deployment (copying that) and you don't 
need to rebuild that all the time.


>
> Cheers,
>
> Ronald
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>
>
>
> _______________________________________________
> Betterform-developer mailing list
> Bet...@li...
> https://lists.sourceforge.net/lists/listinfo/betterform-developer
>

[Betterform-developer] Compressed AND uncompressed version of js?

[Ronald v. K. <Rva...@de...>]

Hi,

Is it in one way or another possible to have both the compressed and uncompressed versions of the javascript in both a exploded deploy and the single jar deploy? And preferably switch by e.g. setting the debug flag in the config?

That way it is fairly easy to switch from develop to production mode. It is e.g. sometimes handy if we can debug on 'preprod' systems without having to create a special build

Cheers,

Ronald

Re: [Betterform-developer] HttpSubmission Error messages get lost...

[Ronald v. K. <Rva...@de...>]

Ok, I will, still getting used to having commit access ;-)

________________________________________
From: Joern Turner [joe...@be...]
Sent: Wednesday, September 12, 2012 10:14 PM
To: Ronald van Kuijk
Cc: bet...@li...; Antal Van Kalleveen
Subject: Re: [Betterform-developer] HttpSubmission Error messages get lost...

Hi Ronald,

sorry for delay - i think you are mainly right with your assumptions.
Nevertheless i come back to my proposal to setup a git branch (branching
development) so you can push such things into that as this makes it much
easier for me to follow just by looking at the diffs. If we find these
are fine we can simply merge in otherwise it does no harm at all (though
i doubt that case really ever happens due to your thorough work ;)

I'll just created a 'contrib' branch and i would suggest that you
*please* put your changes there. Please also feel free to push into that
often even if it's half-baked - that what branches are for. This way i
have a chance to quickly have a look and even contribute my part so
things can quickly go back into 'development'.

cheers,

Joern


Am 07.09.12 17:50, schrieb Ronald van Kuijk:
> Hi,
>
> There seems to be going something wrong in handeling http submission
> error responses from backend systems. The error message that is passed
> on in the body ((reponse-body) gets lost along the way.
>
> The first part of this seems to be in the excecute method in
> AbstractHTTPConnector.java
>
> HttpResponse httpResponse = client.execute(httpRequestBase);
> statusCode = httpResponse.getStatusLine().getStatusCode();
> reasonPhrase = httpResponse.getStatusLine().getReasonPhrase();
> try {
> if ( statusCode >= 300) {
> // Allow 302 only
> if (statusCode != 302) {
> throw new XFormsInternalSubmitException(statusCode, reasonPhrase,
> EntityUtils.toString(httpResponse.getEntity()),
> XFormsConstants.RESOURCE_ERROR);
> }
> }
> this.handleHttpMethod(httpResponse);
> }
> catch (Exception e) {
>
> LOGGER.trace("AbstractHTTPConnector Exception: ", e);
> try {
> throw new
> XFormsInternalSubmitException(httpResponse.getStatusLine().getStatusCode(),
> httpResponse.getStatusLine().getReasonPhrase(),
> EntityUtils.toString(httpResponse.getEntity()),
> XFormsConstants.RESOURCE_ERROR);
> } catch (IOException e1) {
> throw new
> XFormsInternalSubmitException(httpResponse.getStatusLine().getStatusCode(),
> httpResponse.getStatusLine().getReasonPhrase(),
> XFormsConstants.RESOURCE_ERROR);
> }
> }
>
> }
>
>
> Any status code above 300  (except 302) is throwing an exception. When
> creating this exception, the body of the http response is read and
> processed. Al is fine up to here. But immediately after this, this
> exception is caught again and a second attempt to read the body is made.
> This fails (inputstream not available anymore) and a very generic error
> is thrown. This could be changed in several ways.
> - add a 'catch (XFormsException e) { throw e} right before the already
> existing catch
> - change the order of things a little
>
> But anyway... Even if this is corrected, the submit method in
> HTTPSubmissionHandler has a
>
>          } catch (Exception e) {
>              throw new XFormsException(e);
>          }
>
> which wrap (unneededly so) the exception that already is an XFormsException.
>
> This leads to a 'problem' in the submit method the Submission class....
> The following exception handling just after the call to the submit in
> HTTPSubmissionHandler will NOT go into the first catch (the exception is
> wrapped), although that would be the most logical thing to do
>
>         }
>          catch (XFormsInternalSubmitException e) {
>              Map<String, Object> info =
> XFormsSubmitError.constructInfoObject(this.element, this.container,
> locationPath, e.getErrorType(), getResourceURI(), e.getStatusCode(),
> null, e.getStatusText(), e.getResponseBodyAsString());
>              throw new XFormsSubmitError("instance submission failed at:
> " + DOMUtil.getCanonicalPath(this.getElement()), e, this.getTarget(), info);
>          }
>
>
> The second catch fires and since e is NOT an
> XFormsInternalSubmitException (it cannot even be since the catch before
> it should have handled that), so a generic resource-error type is set
> and in the construction of the constructInfoObject, there is no
> reference to whatever body text there was...
>
>          catch (Exception e) {
>              String errorType;
>              if(e instanceof XFormsInternalSubmitException){
>                  errorType =
> ((XFormsInternalSubmitException)e).getErrorType();
>              } else {
>                  errorType = XFormsConstants.RESOURCE_ERROR;
>              }
>              Map<String, Object> info =
> XFormsSubmitError.constructInfoObject(this.element, this.container,
> locationPath, errorType, getResourceURI());
>
>
> The only reference is from the e.getCause().getMessage() (assuming the
> initial fix in AbstractHTTPConnector is applied) returns:
>
> "500 Internal Server Error <our body> resource-error"
>
> Not something you would want to show end-users...
>
>              //todo: hacky - event context info construction must be
> reviewed - using exception cause as response-reason-phrase for now
>              if (e.getCause() != null && e.getCause().getMessage() !=
> null) {
>                  info.put(RESPONSE_REASON_PHRASE,e.getCause().getMessage());
>                  if(e.getCause() instanceof XFormsInternalSubmitException){
>                      info.put(RESPONSE_STATUS_CODE, new
> Integer(((XFormsInternalSubmitException)e.getCause()).getStatusCode()).doubleValue());
>                  }
>              }
>              throw new XFormsSubmitError("instance submission failed at:
> " + DOMUtil.getCanonicalPath(this.getElement()), e, this.getTarget(), info);
>              //throw new XFormsSubmitError("instance submission failed",
> e, this.getTarget(), this.action);
>          }
>
> This can all be fixed by changing the
>
>
>          } catch (Exception e) {
>              throw new XFormsException(e);
>          }
>
> in HTTPSubmissionHandler to
>
> } catch (XFormsException e) {
>              throw e;
>          } catch (Exception e) {
>              throw new XFormsException(e);
>          }
>
> The 'response-status-code' btw seems to be set to 'NaN' but that is a
> minor issue...
>
> Is this analysis valid or did I miss something somewhere...????
>
> Cheers, and have a nice weekend...
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>
>
>
> _______________________________________________
> Betterform-developer mailing list
> Bet...@li...
> https://lists.sourceforge.net/lists/listinfo/betterform-developer
>

Re: [Betterform-developer] Opera - Select1/Combobox missing button

[Joern T. <joe...@be...>]

Am 04.09.12 15:21, schrieb Christian Forster:
> Dear developers,
>
> With the Opera browser, the dropdown button is missing for
> Select1/Combobox in Windows 7, Server 2003 and 2008. I noticed this
> behavior at first in my application based on the version of betterFORM
> that comes with eXist 1.4.2, but it happens also in the online
> demonstration:
>
> http://betterform.de/forms/reference/FeatureExplorer.xhtml Select1 ->
> Combobox
>
> The Linux version of Opera is working fine.
>
> I know that Opera is not in the list of officially tested browsers, so
> please tell me, if you are not interested in such bug reports.
We're always interested and please keep on posting but as you pointed 
out Opera is not on our official list and therefore is unlikely we'll 
have the time fix that ourselves.

But there might be someone out there in the community that has a bit of 
spare time and is fond of Opera that might provide a patch and in that 
case we're happy to integrate that.

Thanks
>
> Best,
> Christian
>

Re: [Betterform-developer] HttpSubmission Error messages get lost...

[Joern T. <joe...@be...>]

Hi Ronald,

sorry for delay - i think you are mainly right with your assumptions. 
Nevertheless i come back to my proposal to setup a git branch (branching 
development) so you can push such things into that as this makes it much 
easier for me to follow just by looking at the diffs. If we find these 
are fine we can simply merge in otherwise it does no harm at all (though 
i doubt that case really ever happens due to your thorough work ;)

I'll just created a 'contrib' branch and i would suggest that you 
*please* put your changes there. Please also feel free to push into that 
often even if it's half-baked - that what branches are for. This way i 
have a chance to quickly have a look and even contribute my part so 
things can quickly go back into 'development'.

cheers,

Joern


Am 07.09.12 17:50, schrieb Ronald van Kuijk:
> Hi,
>
> There seems to be going something wrong in handeling http submission
> error responses from backend systems. The error message that is passed
> on in the body ((reponse-body) gets lost along the way.
>
> The first part of this seems to be in the excecute method in
> AbstractHTTPConnector.java
>
> HttpResponse httpResponse = client.execute(httpRequestBase);
> statusCode = httpResponse.getStatusLine().getStatusCode();
> reasonPhrase = httpResponse.getStatusLine().getReasonPhrase();
> try {
> if ( statusCode >= 300) {
> // Allow 302 only
> if (statusCode != 302) {
> throw new XFormsInternalSubmitException(statusCode, reasonPhrase,
> EntityUtils.toString(httpResponse.getEntity()),
> XFormsConstants.RESOURCE_ERROR);
> }
> }
> this.handleHttpMethod(httpResponse);
> }
> catch (Exception e) {
>
> LOGGER.trace("AbstractHTTPConnector Exception: ", e);
> try {
> throw new
> XFormsInternalSubmitException(httpResponse.getStatusLine().getStatusCode(),
> httpResponse.getStatusLine().getReasonPhrase(),
> EntityUtils.toString(httpResponse.getEntity()),
> XFormsConstants.RESOURCE_ERROR);
> } catch (IOException e1) {
> throw new
> XFormsInternalSubmitException(httpResponse.getStatusLine().getStatusCode(),
> httpResponse.getStatusLine().getReasonPhrase(),
> XFormsConstants.RESOURCE_ERROR);
> }
> }
>
> }
>
>
> Any status code above 300  (except 302) is throwing an exception. When
> creating this exception, the body of the http response is read and
> processed. Al is fine up to here. But immediately after this, this
> exception is caught again and a second attempt to read the body is made.
> This fails (inputstream not available anymore) and a very generic error
> is thrown. This could be changed in several ways.
> - add a 'catch (XFormsException e) { throw e} right before the already
> existing catch
> - change the order of things a little
>
> But anyway... Even if this is corrected, the submit method in
> HTTPSubmissionHandler has a
>
>          } catch (Exception e) {
>              throw new XFormsException(e);
>          }
>
> which wrap (unneededly so) the exception that already is an XFormsException.
>
> This leads to a 'problem' in the submit method the Submission class....
> The following exception handling just after the call to the submit in
> HTTPSubmissionHandler will NOT go into the first catch (the exception is
> wrapped), although that would be the most logical thing to do
>
>         }
>          catch (XFormsInternalSubmitException e) {
>              Map<String, Object> info =
> XFormsSubmitError.constructInfoObject(this.element, this.container,
> locationPath, e.getErrorType(), getResourceURI(), e.getStatusCode(),
> null, e.getStatusText(), e.getResponseBodyAsString());
>              throw new XFormsSubmitError("instance submission failed at:
> " + DOMUtil.getCanonicalPath(this.getElement()), e, this.getTarget(), info);
>          }
>
>
> The second catch fires and since e is NOT an
> XFormsInternalSubmitException (it cannot even be since the catch before
> it should have handled that), so a generic resource-error type is set
> and in the construction of the constructInfoObject, there is no
> reference to whatever body text there was...
>
>          catch (Exception e) {
>              String errorType;
>              if(e instanceof XFormsInternalSubmitException){
>                  errorType =
> ((XFormsInternalSubmitException)e).getErrorType();
>              } else {
>                  errorType = XFormsConstants.RESOURCE_ERROR;
>              }
>              Map<String, Object> info =
> XFormsSubmitError.constructInfoObject(this.element, this.container,
> locationPath, errorType, getResourceURI());
>
>
> The only reference is from the e.getCause().getMessage() (assuming the
> initial fix in AbstractHTTPConnector is applied) returns:
>
> "500 Internal Server Error <our body> resource-error"
>
> Not something you would want to show end-users...
>
>              //todo: hacky - event context info construction must be
> reviewed - using exception cause as response-reason-phrase for now
>              if (e.getCause() != null && e.getCause().getMessage() !=
> null) {
>                  info.put(RESPONSE_REASON_PHRASE,e.getCause().getMessage());
>                  if(e.getCause() instanceof XFormsInternalSubmitException){
>                      info.put(RESPONSE_STATUS_CODE, new
> Integer(((XFormsInternalSubmitException)e.getCause()).getStatusCode()).doubleValue());
>                  }
>              }
>              throw new XFormsSubmitError("instance submission failed at:
> " + DOMUtil.getCanonicalPath(this.getElement()), e, this.getTarget(), info);
>              //throw new XFormsSubmitError("instance submission failed",
> e, this.getTarget(), this.action);
>          }
>
> This can all be fixed by changing the
>
>
>          } catch (Exception e) {
>              throw new XFormsException(e);
>          }
>
> in HTTPSubmissionHandler to
>
> } catch (XFormsException e) {
>              throw e;
>          } catch (Exception e) {
>              throw new XFormsException(e);
>          }
>
> The 'response-status-code' btw seems to be set to 'NaN' but that is a
> minor issue...
>
> Is this analysis valid or did I miss something somewhere...????
>
> Cheers, and have a nice weekend...
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>
>
>
> _______________________________________________
> Betterform-developer mailing list
> Bet...@li...
> https://lists.sourceforge.net/lists/listinfo/betterform-developer
>

[Betterform-developer] HttpSubmission Error messages get lost...

[Ronald v. K. <Rva...@de...>]

Hi,

There seems to be going something wrong in handeling http submission error responses from backend systems. The error message that is passed on in the body ((reponse-body) gets lost along the way.

The first part of this seems to be in the excecute method in AbstractHTTPConnector.java


        HttpResponse httpResponse = client.execute(httpRequestBase);
        statusCode = httpResponse.getStatusLine().getStatusCode();
        reasonPhrase = httpResponse.getStatusLine().getReasonPhrase();
        try {
            if ( statusCode >= 300) {
                // Allow 302 only
                if (statusCode != 302) {
                    throw new XFormsInternalSubmitException(statusCode, reasonPhrase, EntityUtils.toString(httpResponse.getEntity()), XFormsConstants.RESOURCE_ERROR);
                }
            }
            this.handleHttpMethod(httpResponse);
        }
        catch (Exception e) {

            LOGGER.trace("AbstractHTTPConnector Exception: ", e);
            try {
                throw new XFormsInternalSubmitException(httpResponse.getStatusLine().getStatusCode(), httpResponse.getStatusLine().getReasonPhrase(), EntityUtils.toString(httpResponse.getEntity()), XFormsConstants.RESOURCE_ERROR);
            } catch (IOException e1) {
                throw new XFormsInternalSubmitException(httpResponse.getStatusLine().getStatusCode(), httpResponse.getStatusLine().getReasonPhrase(), XFormsConstants.RESOURCE_ERROR);
            }
        }

    }



Any status code above 300  (except 302) is throwing an exception. When creating this exception, the body of the http response is read and processed. Al is fine up to here. But immediately after this, this exception is caught again and a second attempt to read the body is made. This fails (inputstream not available anymore) and a very generic error is thrown. This could be changed in several ways.
- add a 'catch (XFormsException e) { throw e} right before the already existing catch
- change the order of things a little

But anyway... Even if this is corrected, the submit method in HTTPSubmissionHandler has a

        } catch (Exception e) {
            throw new XFormsException(e);
        }

which wrap (unneededly so) the exception that already is an XFormsException.

This leads to a 'problem' in the submit method the Submission class.... The following exception handling just after the call to the submit in HTTPSubmissionHandler will NOT go into the first catch (the exception is wrapped), although that would be the most logical thing to do

        }
        catch (XFormsInternalSubmitException e) {
            Map<String, Object> info = XFormsSubmitError.constructInfoObject(this.element, this.container, locationPath, e.getErrorType(), getResourceURI(), e.getStatusCode(), null, e.getStatusText(), e.getResponseBodyAsString());
            throw new XFormsSubmitError("instance submission failed at: " + DOMUtil.getCanonicalPath(this.getElement()), e, this.getTarget(), info);
        }


The second catch fires and since e is NOT an XFormsInternalSubmitException (it cannot even be since the catch before it should have handled that), so a generic resource-error type is set and in the construction of the constructInfoObject, there is no reference to whatever body text there was...

        catch (Exception e) {
            String errorType;
            if(e instanceof XFormsInternalSubmitException){
                errorType = ((XFormsInternalSubmitException)e).getErrorType();
            } else {
                errorType = XFormsConstants.RESOURCE_ERROR;
            }
            Map<String, Object> info = XFormsSubmitError.constructInfoObject(this.element, this.container, locationPath, errorType, getResourceURI());


The only reference is from the e.getCause().getMessage() (assuming the initial fix in AbstractHTTPConnector is applied) returns:

"500 Internal Server Error <our body> resource-error"

Not something you would want to show end-users...

            //todo: hacky - event context info construction must be reviewed - using exception cause as response-reason-phrase for now
            if (e.getCause() != null && e.getCause().getMessage() != null) {
                info.put(RESPONSE_REASON_PHRASE,e.getCause().getMessage());
                if(e.getCause() instanceof XFormsInternalSubmitException){
                    info.put(RESPONSE_STATUS_CODE, new Integer(((XFormsInternalSubmitException)e.getCause()).getStatusCode()).doubleValue());
                }
            }
            throw new XFormsSubmitError("instance submission failed at: " + DOMUtil.getCanonicalPath(this.getElement()), e, this.getTarget(), info);
            //throw new XFormsSubmitError("instance submission failed", e, this.getTarget(), this.action);
        }

This can all be fixed by changing the


        } catch (Exception e) {
            throw new XFormsException(e);
        }

in HTTPSubmissionHandler to

       } catch (XFormsException e) {
            throw e;
        } catch (Exception e) {
            throw new XFormsException(e);
        }

The 'response-status-code' btw seems to be set to 'NaN' but that is a minor issue...

Is this analysis valid or did I miss something somewhere...????

Cheers, and have a nice weekend...

[Betterform-developer] Opera - Select1/Combobox missing button

[Christian F. <chr...@wi...>]

Dear developers,

With the Opera browser, the dropdown button is missing for
Select1/Combobox in Windows 7, Server 2003 and 2008. I noticed this
behavior at first in my application based on the version of betterFORM
that comes with eXist 1.4.2, but it happens also in the online
demonstration:

http://betterform.de/forms/reference/FeatureExplorer.xhtml Select1 ->
Combobox

The Linux version of Opera is working fine.

I know that Opera is not in the list of officially tested browsers, so
please tell me, if you are not interested in such bug reports.

Best,
Christian

-- 
Diplom-Wirtschaftsinformatiker Christian Forster
Westfälische Wilhelms-Universität Münster
Institut für Wirtschaftsinformatik
Lehrstuhl für Informatik
Leonardo-Campus 3
48149 Münster
Tel.: +49 251 83-38157
http://dbis-group.uni-muenster.de

Re: [Betterform-developer] Consequences of Saxon 9.4 migration [div]

[Ronald v. K. <Rva...@de...>]

> > The exception should be made more specific in Saxon I think (will post
> > an issue there) Now it is a ValidationFailure, but without the
> > "err:FORG0001"  error code.
> >
> > This test needs to be changed I think.
> 
> Hm, i think ths is kind of a problem. Normally in XForms in such cases NaN is
> returned. If this suddenly throws an exception we at least would have to convert
> this to a xforms-binding-exception or xforms-compute-exception but these are
> reserved for certain situations.

A binding exception is difficult since you do not really know what will be the 'toString' of . when creating the binds... 

Catching the xpath exception and throwing an Xforms-compute-exception would be possible, but not sure yet how if can be differentiated from other xpath exceptions. Will look into that. But these will normally show up design time. If you want to use . as a number, you should take care of casting, testing etc... 

> 
> I think such an expression should not raise an exception - at least not from the
> point of view of the user.

Agreed, but this should be dealt with by the forms designer.  See statement above.

Re: [Betterform-developer] Consequences of Saxon 9.4 migration [div]

[Joern T. <joe...@be...>]

Am 10.08.12 14:29, schrieb Ronald van Kuijk:
> Testcase: de.betterform.xml.xforms.ui.OutputTest.java
>
> The following expression does not return NaN anymore but throws an
> exception instead:
>
>              <xf:output id="output-nan-expression-value"
> value="sum(item) div ."/>
>
> The exception should be made more specific in Saxon I think (will post
> an issue there) Now it is a ValidationFailure, but without the
> “err:FORG0001”  error code.
>
> This test needs to be changed I think.

Hm, i think ths is kind of a problem. Normally in XForms in such cases 
NaN is returned. If this suddenly throws an exception we at least would 
have to convert this to a xforms-binding-exception or 
xforms-compute-exception but these are reserved for certain situations.

I think such an expression should not raise an exception - at least not 
from the point of view of the user.

>
> Ronald
>
> *From:*Ronald van Kuijk [mailto:Rva...@de...]
> *Sent:* vrijdag 10 augustus 2012 14:25
> *To:* bet...@li...
> *Subject:* [Betterform-developer] Consequences of Saxon 9.4 migration
>
> This thread will be used to communicate about the changes I found that
> resulted from migrating to Saxon 9.4. I will make each ‘issue’ a child
> of this message… and a suffix in the subject
>
> Cheers,
>
> Ronald
>
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>
>
>
> _______________________________________________
> Betterform-developer mailing list
> Bet...@li...
> https://lists.sourceforge.net/lists/listinfo/betterform-developer
>

Re: [Betterform-developer] Consequences of Saxon 9.4 migration [div]

[Ronald v. K. <Rva...@de...>]

Testcase: de.betterform.xml.xforms.ui.OutputTest.java

The following expression does not return NaN anymore but throws an exception instead:

            <xf:output id="output-nan-expression-value" value="sum(item) div ."/>

The exception should be made more specific in Saxon I think (will post an issue there) Now it is a ValidationFailure, but without the "err:FORG0001"  error code.

This test needs to be changed I think.

Ronald


From: Ronald van Kuijk [mailto:Rva...@de...]
Sent: vrijdag 10 augustus 2012 14:25
To: bet...@li...
Subject: [Betterform-developer] Consequences of Saxon 9.4 migration

This thread will be used to communicate about the changes I found that resulted from migrating to Saxon 9.4. I will make each 'issue' a child of this message... and a suffix in the subject

Cheers,

Ronald

[Betterform-developer] Consequences of Saxon 9.4 migration

[Ronald v. K. <Rva...@de...>]

This thread will be used to communicate about the changes I found that resulted from migrating to Saxon 9.4. I will make each 'issue' a child of this message... and a suffix in the subject

Cheers,

Ronald

Re: [Betterform-developer] [Betterform-users] sum function behaviour 'issue'

[Joern T. <joe...@be...>]

Am 03.08.12 13:33, schrieb Ronald van Kuijk:
> (cc to the developer list)
>
> Well, I think, no am SURE I found the cause of this... Saxon has the
> concept of a FunctionLibrary that can be used to define all kinds of
> functions in all kinds of namespaces in an extensible way.
>
> It seemed that the sum function I was calling was in the xforms
> namespace. Ok, no problem, but the actual implementation being called
> was NOT the xpath2 Sum from Saxon.  Many of the functions that are
> available in the xpath namespace have been 'mapped' to the xforms
> namespace (call the actual saxon implementation) but not all.
>
> The actual implementation that is being called is defined in
> XFormsFunctionLibrary
>
> e = register("{" + NamespaceConstants.XFORMS_NS + "}sum",
> Aggregate2.class, Aggregate2.SUM, 1, 2, BuiltInAtomicType.ANY_ATOMIC,
> StaticProperty.ALLOWS_ZERO_OR_ONE); arg(e, 0,
> BuiltInAtomicType.ANY_ATOMIC, StaticProperty.ALLOWS_ZERO_OR_MORE,
> null); arg(e, 1, BuiltInAtomicType.ANY_ATOMIC,
> StaticProperty.ALLOWS_ZERO_OR_ONE, null);
>
> The Aggregate2 class serves the implementation and when looking at
> that, it is just, well, not fully behaving like I think it should if
> the behavior should be according to XPath2 (not checked XPath1). The
> 'total' method checks if an empty sequence is encountered and if so,
> immediately returns DoubleValue.NaN. This return value is then
> checked to see if it is null and if so returns either 0 or the second
> parameter if present. But since the return value is never null, it
> just does not work.
>
> It can (should?) be fixed in Aggregate2, but it can also be solved by
> pointing the sum function to the Saxon Sum.class. I checked this and
> it still works. For a kind of backwards compatibility, the current
> sum function could not fixed and in name be changed to SUM (like done
> with if) and the
i think it would be perfectly ok to map to the saxon implementation. I 
even do not see much of a problem in NOT keeping the older one (if it's 
somehow broken anyway)

>
> But this leads to a more fundamental question. Can we use all
> functions from XPath2, including the ones that are not mapped to the
> xforms namespace
As far as i remember we can't. To be able to use XPath 2 functions in 
XForms without prefixing (namespacing) we need to do this mapping. It 
makes sure that the XPath functions are available as 'XForms builtin 
functions'.  For XForms 1.1 this is not strictly correct as it still 
uses XPath 1 but again i do not see a problem in doing so - XPath 2 will 
be part of XForms 2.0 anyway and as far as i can see XPath 2 is a 
superset of XPath 1 and not overwriting any behavior of XPath 1 - there 
might be nifty cases where it makes a difference but i tend to ignore 
those for the sake of simplicity - nobody will want to continue using 
XPath 1 if XPath 2 is there - if anybody disagrees please shout out now!


>
> I tried explicitly using the xpath2 namespace (after making it
> available as a prefix)
>
> assertEquals(0d,
> evaluateInDefaultContextAsDouble("fn:sum((/non-existing))"));
>
> but then I get an error:
>
> Caused by: net.sf.saxon.trans.XPathException: System function sum#1
> is not available with this host language
>
> A little debugging showed that saxon internally  things it is not
> ALLOWED to use this function... Will need to check this out.
>
> So it seems we are currently 'limited'...
>
> Cheers and have a nice weekend...
>
> Ronald
>
>
> ------------------------------------------------------------------------------
>
>
Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond.
> Discussions will include endpoint security, mobile security and the
> latest in malware threats.
> http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________ Betterform-users
> mailing list Bet...@li...
> https://lists.sourceforge.net/lists/listinfo/betterform-users
>
>

Re: [Betterform-developer] [Betterform-users] sum function behaviour 'issue'

[Joern T. <joe...@be...>]

Am 06.08.12 09:58, schrieb Ronald van Kuijk:
> But more important... can you guys make a statement on the behaviour of sum? That is important for me.

Well yes, i try ;)

As it seems to me the behavior of sum for empty nodes is correct as it 
uses the number() function to convert the node values before summing up. 
However number() returns NaN whenever the node value cannot be used as a 
number.

I'm still not sure if i understood the use case right - (sorry haven't 
read through the whole thread yet ;)

As i understand there might be empty nodes within your nodeset that make 
your summing crash.

You already used the expression below to address this - right?

sum(/item[number(child) = child]/child)

I found another slightly more improved variant that also addresses the 
issue of empty nodes. Would be like this:
sum(/item[number(child) = number(child)]/child)

I just copy the original explanation....
"number($v) = number($v) is obviously true, if $v is a number, or a 
string that represents a number.

It is true also for a boolean value, because a number(true()) is 1 and 
number(false) is 0.

Whenever $v cannot be used as a number, then number($v) is NaN

and NaN is not equal to any other value, even to itself.

Thus, the above expression is true only for $v whose value can be used 
as a number, and false otherwise."

So it's probably not necessary to have nested if then else construct 
which are surely terrible.

Did i miss something?




>
>> -----Original Message-----
>> From: Lars Windauer [mailto:lar...@be...]
>> Sent: zaterdag 4 augustus 2012 1:33
>> To: Ronald van Kuijk
>> Cc: 'Thorsten Trippel'; bet...@li...; betterform-
>> dev...@li...
>> Subject: Re: [Betterform-users] sum function behaviour 'issue'
>>
>> Hej Ronald,
>>
>> Great news! Sorry we are such quiet at the moment but we have lot's of stuff
>> going on and great stuff to come up with in the near future  By the way, one thing
>> on our list is to switch to Saxon 9.3 (and use  a different API) to take advantage of
>> the commercial Saxon versions in customer projects.
>>
>> Thanks for your work, looking forward to you patch ;-)
>>
>> best regards
>>
>> Lars
>>
>>
>>
>> On 03.08.2012, at 17:52, Ronald van Kuijk <Rva...@de...> wrote:
>>
>>>
>>>>
>>>> p.s. Looks like most of the interesting new things in xpath2 are not
>>>> in functions, so the 'mapping' to the xforms namespace seems to fairly
>> complete...
>>>>
>>>
>>> [Ronald van Kuijk]
>>> I found a simple solution to enable all functions in the fn: namespace... Will make
>> a patch for this.
>>>
>>> ----------------------------------------------------------------------
>>> --------
>>> Live Security Virtual Conference
>>> Exclusive live event will cover all the ways today's security and
>>> threat landscape has changed and how IT managers can respond.
>>> Discussions will include endpoint security, mobile security and the
>>> latest in malware threats.
>>> http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>> _______________________________________________
>>> Betterform-users mailing list
>>> Bet...@li...
>>> https://lists.sourceforge.net/lists/listinfo/betterform-users
>>>
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Betterform-developer mailing list
> Bet...@li...
> https://lists.sourceforge.net/lists/listinfo/betterform-developer
>
>

Re: [Betterform-developer] [Betterform-users] sum function behaviour 'issue'

[Ronald v. K. <Rva...@de...>]

But more important... can you guys make a statement on the behaviour of sum? That is important for me. 

> -----Original Message-----
> From: Lars Windauer [mailto:lar...@be...]
> Sent: zaterdag 4 augustus 2012 1:33
> To: Ronald van Kuijk
> Cc: 'Thorsten Trippel'; bet...@li...; betterform-
> dev...@li...
> Subject: Re: [Betterform-users] sum function behaviour 'issue'
> 
> Hej Ronald,
> 
> Great news! Sorry we are such quiet at the moment but we have lot's of stuff
> going on and great stuff to come up with in the near future  By the way, one thing
> on our list is to switch to Saxon 9.3 (and use  a different API) to take advantage of
> the commercial Saxon versions in customer projects.
> 
> Thanks for your work, looking forward to you patch ;-)
> 
> best regards
> 
> Lars
> 
> 
> 
> On 03.08.2012, at 17:52, Ronald van Kuijk <Rva...@de...> wrote:
> 
> >
> >>
> >> p.s. Looks like most of the interesting new things in xpath2 are not
> >> in functions, so the 'mapping' to the xforms namespace seems to fairly
> complete...
> >>
> >
> > [Ronald van Kuijk]
> > I found a simple solution to enable all functions in the fn: namespace... Will make
> a patch for this.
> >
> > ----------------------------------------------------------------------
> > --------
> > Live Security Virtual Conference
> > Exclusive live event will cover all the ways today's security and
> > threat landscape has changed and how IT managers can respond.
> > Discussions will include endpoint security, mobile security and the
> > latest in malware threats.
> > http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> > _______________________________________________
> > Betterform-users mailing list
> > Bet...@li...
> > https://lists.sourceforge.net/lists/listinfo/betterform-users
> >

Re: [Betterform-developer] [Betterform-users] sum function behaviour 'issue'

[Ronald v. K. <Rva...@de...>]

No problem being busy...  :-)

Switching to Saxon 9.3 by itself was not that difficult. Other advantage was that some duplicated code could be removed. What I also like to implement is loading custom functions... There is a new mechanism in Saxon (since 9.2) that makes it way more simple to add custom functions. No more BetterFormFunctionLibrary etc... and declaring all stuff in the implementation class

http://www.saxonica.com/documentation/extensibility/integratedfunctions/ext-full-J.xml

what could be easily done is add custom extension functions to a config file since all that is needed is loading such a function via reflection and instantiating it

        	Class clazz = cl.loadClass("de.betterform.xml.xforms.xpath.saxon.function.xpath.ShiftLeft");
        	ExtensionFunctionDefinition myObj = (ExtensionFunctionDefinition) clazz.newInstance();
			XPathCache.kCONFIG.registerExtensionFunction(myObj);


And the end one line:

        fgXFormsFunctionLibrary.addFunctionLibrary(XPathCache.kCONFIG.getIntegratedFunctionLibrary());

Would also be a nice addition I think...

> -----Original Message-----
> From: Lars Windauer [mailto:lar...@be...]
> Sent: zaterdag 4 augustus 2012 1:33
> To: Ronald van Kuijk
> Cc: 'Thorsten Trippel'; bet...@li...; betterform-
> dev...@li...
> Subject: Re: [Betterform-users] sum function behaviour 'issue'
> 
> Hej Ronald,
> 
> Great news! Sorry we are such quiet at the moment but we have lot's of stuff
> going on and great stuff to come up with in the near future  By the way, one thing
> on our list is to switch to Saxon 9.3 (and use  a different API) to take advantage of
> the commercial Saxon versions in customer projects.
> 
> Thanks for your work, looking forward to you patch ;-)
> 
> best regards
> 
> Lars
> 
> 
> 
> On 03.08.2012, at 17:52, Ronald van Kuijk <Rva...@de...> wrote:
> 
> >
> >>
> >> p.s. Looks like most of the interesting new things in xpath2 are not
> >> in functions, so the 'mapping' to the xforms namespace seems to fairly
> complete...
> >>
> >
> > [Ronald van Kuijk]
> > I found a simple solution to enable all functions in the fn: namespace... Will make
> a patch for this.
> >
> > ----------------------------------------------------------------------
> > --------
> > Live Security Virtual Conference
> > Exclusive live event will cover all the ways today's security and
> > threat landscape has changed and how IT managers can respond.
> > Discussions will include endpoint security, mobile security and the
> > latest in malware threats.
> > http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> > _______________________________________________
> > Betterform-users mailing list
> > Bet...@li...
> > https://lists.sourceforge.net/lists/listinfo/betterform-users
> >

Re: [Betterform-developer] [Betterform-users] sum function behaviour 'issue'

[Lars W. <lar...@be...>]

Hej Ronald, 

Great news! Sorry we are such quiet at the moment but we have lot's of stuff going on and great stuff to come up with in the near future  By the way, one thing on our list is to switch to Saxon 9.3 (and use  a different API) to take advantage of the commercial Saxon versions in customer projects. 

Thanks for your work, looking forward to you patch ;-)

best regards

Lars


 
On 03.08.2012, at 17:52, Ronald van Kuijk <Rva...@de...> wrote:

> 
>> 
>> p.s. Looks like most of the interesting new things in xpath2 are not in functions, so
>> the 'mapping' to the xforms namespace seems to fairly complete...
>> 
> 
> [Ronald van Kuijk] 
> I found a simple solution to enable all functions in the fn: namespace... Will make a patch for this.
> 
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and 
> threat landscape has changed and how IT managers can respond. Discussions 
> will include endpoint security, mobile security and the latest in malware 
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Betterform-users mailing list
> Bet...@li...
> https://lists.sourceforge.net/lists/listinfo/betterform-users
> 

Re: [Betterform-developer] [Betterform-users] sum function behaviour 'issue'

[Ronald v. K. <Rva...@de...>]

> 
> p.s. Looks like most of the interesting new things in xpath2 are not in functions, so
> the 'mapping' to the xforms namespace seems to fairly complete...
> 

[Ronald van Kuijk] 
I found a simple solution to enable all functions in the fn: namespace... Will make a patch for this.

Re: [Betterform-developer] [Betterform-users] sum function behaviour 'issue'

[Ronald v. K. <Rva...@de...>]

Ok, there is one little difference in behavior: 

The 'current' Aggregate2 implementation results in a NaN if a sum over a sequence is done where an item contains non numeric value. The Sum from Saxon throws an exception. 

Not sure what the spec says about this, but it can easily be worked around (as it should I think) by adding a check

So 

        assertEquals(0d, evaluateInDefaultContextAsDouble("sum(/data[number(text) = text]/text)")); 

works, but

        assertEquals(Double.NaN, evaluateInDefaultContextAsDouble("sum(/data/text)"));

throws an exception. This latter works in Aggregate2 (the current implementation)

Ronald

p.s. Looks like most of the interesting new things in xpath2 are not in functions, so the 'mapping' to the xforms namespace seems to fairly complete...




> -----Original Message-----
> From: Ronald van Kuijk
> Sent: vrijdag 3 augustus 2012 13:34
> To: Ronald van Kuijk; 'Thorsten Trippel'; bet...@li...
> Cc: bet...@li...
> Subject: RE: [Betterform-users] sum function behaviour 'issue'
> 
> (cc to the developer list)
> 
> Well, I think, no am SURE I found the cause of this... Saxon has the concept of a
> FunctionLibrary that can be used to define all kinds of functions in all kinds of
> namespaces in an extensible way.
> 
> It seemed that the sum function I was calling was in the xforms namespace. Ok,
> no problem, but the actual implementation being called was NOT the xpath2 Sum
> from Saxon.  Many of the functions that are available in the xpath namespace have
> been 'mapped' to the xforms namespace (call the actual saxon implementation)
> but not all.
> 
> The actual implementation that is being called is defined in XFormsFunctionLibrary
> 
>         e = register("{" + NamespaceConstants.XFORMS_NS + "}sum",
> Aggregate2.class, Aggregate2.SUM, 1, 2, BuiltInAtomicType.ANY_ATOMIC,
> StaticProperty.ALLOWS_ZERO_OR_ONE);
>         arg(e, 0, BuiltInAtomicType.ANY_ATOMIC,
> StaticProperty.ALLOWS_ZERO_OR_MORE, null);
>         arg(e, 1, BuiltInAtomicType.ANY_ATOMIC,
> StaticProperty.ALLOWS_ZERO_OR_ONE, null);
> 
> The Aggregate2 class serves the implementation and when looking at that, it is
> just, well, not fully behaving like I think it should if the behavior should be
> according to XPath2 (not checked XPath1).
> The 'total' method checks if an empty sequence is encountered and if so,
> immediately returns DoubleValue.NaN. This return value is then checked to see if it
> is null and if so returns either 0 or the second parameter if present. But since the
> return value is never null, it just does not work.
> 
> It can (should?) be fixed in Aggregate2, but it can also be solved by pointing the
> sum function to the Saxon Sum.class. I checked this and it still works. For a kind of
> backwards compatibility, the current sum function could not fixed and in name be
> changed to SUM (like done with if) and the
> 
> But this leads to a more fundamental question. Can we use all functions from
> XPath2, including the ones that are not mapped to the xforms namespace
> 
> I tried explicitly using the xpath2 namespace (after making it available as a prefix)
> 
> 	assertEquals(0d, evaluateInDefaultContextAsDouble("fn:sum((/non-
> existing))"));
> 
> but then I get an error:
> 
> 	Caused by: net.sf.saxon.trans.XPathException: System function sum#1 is
> not available with this host language
> 
> A little debugging showed that saxon internally  things it is not ALLOWED to use
> this function... Will need to check this out.
> 
> So it seems we are currently 'limited'...
> 
> Cheers and have a nice weekend...
> 
> Ronald

Re: [Betterform-developer] [Betterform-users] sum function behaviour 'issue'

[Ronald v. K. <Rva...@de...>]

(cc to the developer list) 

Well, I think, no am SURE I found the cause of this... Saxon has the concept of a FunctionLibrary that can be used to define all kinds of functions in all kinds of namespaces in an extensible way. 

It seemed that the sum function I was calling was in the xforms namespace. Ok, no problem, but the actual implementation being called was NOT the xpath2 Sum from Saxon.  Many of the functions that are available in the xpath namespace have been 'mapped' to the xforms namespace (call the actual saxon implementation) but not all.

The actual implementation that is being called is defined in XFormsFunctionLibrary

        e = register("{" + NamespaceConstants.XFORMS_NS + "}sum", Aggregate2.class, Aggregate2.SUM, 1, 2, BuiltInAtomicType.ANY_ATOMIC, StaticProperty.ALLOWS_ZERO_OR_ONE);
        arg(e, 0, BuiltInAtomicType.ANY_ATOMIC, StaticProperty.ALLOWS_ZERO_OR_MORE, null);
        arg(e, 1, BuiltInAtomicType.ANY_ATOMIC, StaticProperty.ALLOWS_ZERO_OR_ONE, null);

The Aggregate2 class serves the implementation and when looking at that, it is just, well, not fully behaving like I think it should if the behavior should be according to XPath2 (not checked XPath1). 
The 'total' method checks if an empty sequence is encountered and if so, immediately returns DoubleValue.NaN. This return value is then checked to see if it is null and if so returns either 0 or the second parameter if present. But since the return value is never null, it just does not work. 

It can (should?) be fixed in Aggregate2, but it can also be solved by pointing the sum function to the Saxon Sum.class. I checked this and it still works. For a kind of backwards compatibility, the current sum function could not fixed and in name be changed to SUM (like done with if) and the 

But this leads to a more fundamental question. Can we use all functions from XPath2, including the ones that are not mapped to the xforms namespace

I tried explicitly using the xpath2 namespace (after making it available as a prefix)

	assertEquals(0d, evaluateInDefaultContextAsDouble("fn:sum((/non-existing))"));

but then I get an error:

	Caused by: net.sf.saxon.trans.XPathException: System function sum#1 is not available with this host language

A little debugging showed that saxon internally  things it is not ALLOWED to use this function... Will need to check this out.

So it seems we are currently 'limited'... 

Cheers and have a nice weekend...

Ronald